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Overview 


PACE Center for Girls is a Florida-based organization that provides academic and social services to 
girls of middle school and high school age. Girls who attend PACE have a specific set of characteris- 
tics that put them at risk of involvement with the juvenile justice system and other negative outcomes. 
PACE seeks to reduce the likelihood of negative outcomes tied to this profile and instead foster aca- 
demic engagement, positive youth development, and healthy relationships. Such factors can help im- 
prove girls’ academic outcomes, prevent their future involvement with the justice system, and lead to 
long-term well-being and success. 


PACE operates daily, year-round; ona typical day, girls attend academic classes and receive additional 
support such as individual counseling, academic advising, and referrals to other services. Throughout 
its delivery of these services, PACE uses principles of gender-responsive programming — that is, 
treatment approaches designed for girls and women. 


The current report focuses mainly on the impact and cost analyses for the PACE evaluation. (Imple- 
mentation research findings, released in an earlier report, found that PACE consistently implemented 
its program model and incorporated gender-responsive programming across its centers.) The impact 
analysis employed a random assignment design: Girls who applied to and were deemed eligible for 
PACE (using the program’s existing screening processes) enrolled in the study and were assigned at 
random either to a program group, whose members were offered PACE services, or to a control group, 
whose members received appropriate referrals to other services in the community. From August 2013 
to November 2015, 1,125 girls enrolled in the study across 14 PACE centers. Using survey and ad- 
ministrative data, the research team measured differences between the program and control groups on 
short-term outcomes. Differences that emerge between the two groups on these outcomes can be at- 
tributed to the PACE program. 


Key Findings 


e The program group received more academic and social services — and received them more often 
from a professional source — than the control group. 


e Overa one-year period, PACE increased school enrollment and attendance for the girls it served, 
compared with the control group. Girls in the program group were also more likely to be “on 
track” academically than those in the control group. 


e  Gzrls in both the program and control groups appeared goal-oriented and hopeful about their fu- 
tures and reported relatively low levels of risky behavior one year after study enrollment. Rates 
of formal involvement in the juvenile justice system during the 18 months after study enrollment 
were similar for the program and control groups. 


e Thecost of PACE’s holistic package of services is, on average, $10,400 more than the cost of the 
services received by control group members through academic and social services provided in the 
community. The additional cost is largely driven by PACE’s extensive social services; the cost of 
academic services is similar to those of Florida public schools. 


The findings on academic outcomes are promising. Further follow-up research would be necessary to 
see whether PACE affects longer-term academic and delinquency outcomes and to complete a full 
benefit-cost analysis. 
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Preface 


Practitioners and policymakers constantly look for ways to better support young people who are 
at risk of involvement with the justice system. Young women who fit this profile often face chal- 
lenges and have histories distinct from those of their male counterparts. Yet most programs and 
services targeting at-risk young people are not designed to address issues uniquely experienced 
by girls. Gender-responsive programming was developed in recognition of this fact. There is in- 
terest at the local, state, and national levels in understanding how these types of programs are 
implemented and obtaining more robust evidence on their effectiveness. Preventing future justice 
system involvement and engaging girls in their academic future, or other positive outcomes, can 
have meaningful benefits for an individual girl’s future, her family, and her community. 


MDRC’s evaluation of the PACE Center for Girls offers an opportunity to understand 
the effectiveness of a well-known gender-responsive program. PACE serves girls ages 11 to 18 
at 21 locations across the state of Florida. Girls attend PACE during school hours, year-round, 
and receive academic and social services. Through its program model, PACE aims to increase 
the presence of protective factors for the girls it serves — that is, characteristics or conditions that 
may moderate risk for negative outcomes. Its gender-responsive programming specifically seeks 
to cultivate girls’ strengths, positive interpersonal relationships, and relevant life skills and to en- 
courage other protective factors, such as being connected to and engaged in school. 


An earlier report, focused on the evaluation’s implementation study, detailed PACE’s 
gender-responsive program model and found that PACE implemented the model consistently 
across its many locations. The current report presents short-term impact findings and a cost anal- 
ysis. The impact study, which employed a random assignment design with two research groups, 
found that PACE positively affected girls’ academic engagement and progress. In other domains 
— youth development, risky behavior, and delinquency — girls in both the program and control 
groups had similar outcomes. The cost analysis found that the additional cost of attending PACE, 
compared with the academic and social services the control group received, is principally driven 
by the extensive social services PACE provides. 


The short-term findings are positive and encouraging and will contribute to the growing 
literature on the effectiveness of gender-responsive programming. A longer period of research 
could reveal whether the impacts on short-term academic outcomes lead to long-term outcomes 
such as on-time grade promotion, high school graduation, and preventing justice system involve- 
ment — results that could make PACE truly cost effective. 


Gordon L. Berlin 
President, MDRC 
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Executive Summary 


PACE Center for Girls is a Florida-based nonprofit organization that serves girls of middle school 
and high school age who are at risk of involvement with the justice system. These girls are often 
struggling academically, and some have been involved with the juvenile justice system already. 
They may have mental health issues, often stemming from experiences of trauma, or may engage 
in behavior that negatively affects their physical health. Using a ““gender-responsive” framework 
for its academic and social services, PACE aims to reduce the likelihood of negative outcomes 
tied to this set of risk factors and instead foster academic engagement, positive youth develop- 
ment, and healthy relationships. Increasing these protective factors can improve girls’ academic 
outcomes, prevent their future involvement with the justice system, and lead to long-term well- 
being and success. This report presents findings from a rigorous evaluation of PACE. 


Improving academic outcomes in the short term may help girls get back on track and 
graduate on time. Preventing involvement in the juvenile justice system 1s also crucial, for several 
reasons. Juvenile justice involvement can damage a young person’s relationships with friends and 
family, negatively affect mental health, and interrupt the academic progress and work experience 
that should accumulate during adolescence. ! 


Investing in the well-being of girls promises economic benefits as well. Compared with 
dropouts, high school graduates have been shown to make larger economic contributions to soci- 
ety through lower unemployment rates, higher earnings, more taxes paid, and less reliance on 
public assistance. One estimate found that high school graduates make nearly $300,000 more than 
those who do not complete high school over the course of their lifetimes.” And from a societal 
perspective, the court and detainment costs associated with juvenile justice involvement are high. 
Therefore, effective prevention or early intervention programs can offer a significant return on 
investment.? 


PACE uses principles of gender-responsive programming throughout its services for 
girls. The term “gender-responsive” describes treatment approaches designed for girls and 
women, specifically those involved with or at risk of involvement with the juvenile and criminal 
justice systems. Historically, services for those involved with the justice system were designed 
for boys and men.* Gender-responsive services bring an awareness of girls’ distinctive develop- 
ment patterns and gender-specific issues into the program. While there has been national interest 


‘Anna Aizer and Joseph J. Doyle Jr., “Juvenile Incarceration, Human Capital, and Future Crime: Evidence 
from Randomly Assigned Judges,” Quarterly Journal of Economics 130, no. 2 (2015): 759-803. 

Christopher R. Tamborini, ChangHwan Kim, and Arthur Sakamoto, “Education and Lifetime Earnings in 
the United States,” Demography 52, no. 4 (2015): 1383-1407. This calculation compares lifetime earnings for 
high school dropouts with earnings for high school graduates with no further education. 

3Steve Aos, Roxanne Lieb, Jim Mayfield, Marna Miller, and Annie Pennucci, Benefits and Costs of Pre- 
vention and Early Intervention Programs for Youth (Olympia: Washington State Institute for Public Policy, 
2004). 

4Stephanie Covington and Barbara Bloom, “Center for Gender and Justice” (2017), http://centerforgender 
andjustice.org; Margaret A. Zahn, Stephanie R. Hawkins, Janet Chiancone, and Ariel Whitworth, The Girls Study 
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in understanding gender-responsive programs more broadly,* the current literature is more robust 
in its description of concepts and principles than in its evaluation of program performance.° Until 
recently, it was largely unknown how such services are implemented, how similar they are to one 
another, or how effective they are.’ 


The evaluation of PACE Center for Girls — among the largest and most well-established 
programs of its kind — provides an opportunity to answer questions about the implementation 
and effectiveness of a gender-responsive program. The research aims to help practitioners, in- 
cluding PACE, and policymakers better understand, improve, and possibly replicate services for 
at-risk girls. The implementation research findings, released in an earlier report, found that PACE 
consistently implemented its program model and incorporated gender-responsive programming 
across its statewide network of centers.* The model was defined through both general program 
principles and a detailed program manual. 


The current report shows that PACE helped girls stay more engaged in school and on 
track toward high school graduation during a one-year follow-up period. Across the study sample, 
girls seemed relatively stable one year after enrollment, and hopeful for the future. PACE did not 
affect formal involvement in the justice system over the 18 months following random assignment. 


The evaluation was conducted by MDRC and has been funded mainly through the Edna 
McConnell Clark Foundation’s Social Innovation Fund, a program of the Corporation for Na- 
tional and Community Service, with additional funding provided by the Jessie Ball duPont Fund 
and the Healy Foundation. 


About PACE Center for Girls 


PACE Center for Girls currently operates 21 nonresidential, year-round centers across the state 
of Florida. PACE serves girls between the ages of 11 and 18 who exhibit risk factors across mul- 
tiple domains that are correlated with delinquency in girls. Girls in this voluntary program attend 
PACE daily during normal school hours and receive academic and social services: comprehen- 
sive assessment and care planning, academic instruction and advising, a life skills curriculum, 


Group. Charting the Way to Delinquency Prevention for Girls (Washington, DC: Office of Juvenile Justice and 
Delinquency Prevention, U.S. Department of Justice, 2008). 

Juvenile Justice and Delinquency Prevention Act, 42 U.S.C. 5633 § 242 (1992). 

Dana Jones Hubbard and Betsy Matthews, “Reconciling the Differences Between the ‘Gender-Responsive’ 
and ‘What Works’ Literatures to Improve Services for Girls,” Crime and Delinquency 54, no. 2 (2008): 225- 
258. 

™Meda Chesney-Lind, Merry Morash, and Tia Stevens, “Girls’ Troubles, Girls’ Delinquency, and Gender 
Responsive Programming: A Review,” Australian and New Zealand Journal of Criminology 4, no. 1 (2008): 
162-189; Patricia K. Kerig and Sheryl R. Schindler, “Engendering the Evidence Base: A Critical Review of the 
Conceptual and Empirical Foundations of Gender-Responsive Interventions for Girls’ Delinquency,” Laws 2, 
no. 3 (2013): 244-282. 

8Louisa Treskon, Megan Millenky, and Lily Freedman, Helping Girls Get Back on Track (New York: 
MDRC, 2017). 
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individual and group counseling, volunteer service and work readiness opportunities, and transi- 
tion and follow-up services. Girls typically plan to attend PACE for about one year and often 
return to other schools in their communities to complete their education.? 


As a gender-responsive program, PACE centers strive to create inclusive environments 
in which the support services “wrap around” each girl, and they rely on a strengths-based ap- 
proach — emphasizing a girl’s assets rather than deficits — and an understanding of trauma and 
its effects when dealing with girls’ risky or challenging behaviors. A low staff-to-girl ratio allows 
for individual attention and opportunities to build relationships, contributing to the girls’ sense of 
safety and belonging while they are in attendance. Staff members hold regular reviews to discuss 
each girl’s care plan and progress and emphasize parental engagement, contacting a parent or 
guardian monthly. 


During a typical day at PACE, girls attend classes, usually language arts, math, social stud- 
ies, life skills, and science. Every other week, girls attend counseling sessions that focus on fostering 
positive behavioral change and separate academic advising sessions where staff members monitor 
their progress. Counselors are available more frequently, if needed, and can provide referrals for 
additional supportive or therapeutic services. Once a girl makes the transition to a different school, 
PACE staff members follow up periodically and can continue to connect her to services. 


Figure ES.1 illustrates the basics of the PACE program model and the expected short- 
term outcomes. Participation in PACE’s holistic program is intended to increase a girl’s academic 
engagement and progress and bolster her confidence and interpersonal skills, reducing risky be- 
havior and providing the means for a more positive future outlook. 


The PACE Evaluation 


The evaluation of PACE’s program model presents an opportunity to test the effectiveness of a 
gender-responsive program and provide rigorous evidence on services for girls at risk of negative 
outcomes. The evaluation has three main components: an impact study, an implementation study, 
and a cost analysis. This report focuses mainly on the impact and cost analyses. 


The impact study employs a random assignment design. With this design, girls eligible 
for PACE were enrolled in the study and assigned at random either to a program group, whose 
members were offered PACE services, or to a control group, whose members were referred to 
other academic and social services in the community. Because random assignment resulted in 
two groups with similar observed and unobserved characteristics, any differences in outcomes 
between the two groups can be attributed to the PACE program. 


Fourteen PACE centers participated in the evaluation. Between August 2013 and Novem- 
ber 2015, 1,125 girls were enrolled in the study (673 in the program group and 452 in the control 


°In some cases, girls seek options other than returning to the public school they attended previously or an- 
other school in the district; for example, earning a high school equivalency diploma and gaining employment. In 
some cases, PACE centers provide a high school diploma through the local school district. 
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Figure ES.1 
PACE Program Model 
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group). Most girls were low-income, most were ages 13 to 16 at the time of study enrollment, and 
just over half the sample came from single-parent households. Many sample members struggled 
with school before coming to PACE. At the time of study enrollment, 40 percent had been recently 
expelled or suspended from school, and just over half had been held back at least once. Approxi- 
mately 30 percent of girls had been arrested at some point, and two-thirds of the study sample had 
a family member with a criminal history. Nearly 40 percent of girls in the study reported having 
been abused or neglected. Many of the risk factors the study participants exhibited are among those 
cited in the literature on adverse childhood experiences (ACEs). Research links these experiences 
to future risky behavior, health problems, and early death, as well as other negative outcomes. !° 


After examining the difference between the services both groups participated in during 
this follow-up period, the impact analysis measures effects on short-term outcomes, including 
academic engagement and progress, juvenile justice involvement, risky behaviors, and interper- 
sonal relationships. 


While the evaluation uses a rigorous design and draws on a range of quantitative and 
qualitative data sources, it has some limitations.!! The evaluation’s funding allowed for data col- 
lection for only 12 to 18 months following the girls’ application to PACE and enrollment in the 
study. While this time period provides information on PACE’s short-term outcomes, it was not 
enough time for most girls in the study sample to reach key milestones such as high school grad- 
uation. Therefore, the evaluation does not include a full benefit-cost analysis at this time. Addi- 
tionally, the follow-up survey asked sample members questions that assess risky behavior and 
interpersonal relationships, but these are difficult constructs to measure, especially in a short sur- 
vey usually administered by phone. Among other things, girls may be hesitant to provide honest 
or complete answers to a survey interview about ongoing issues or risky behaviors. Finally, this 
study also benefited from access to Florida’s Department of Education records, but the study team 
had access only to deidentified data, which limited the analyses that could be performed. 


Key Findings 


e The program group received more academic and social services — and re- 
ceived them more often from a professional source — than the control group. 


With PACE’s comprehensive model, girls in the program group received more services 
than girls in the control group in the year following study enrollment. As Figure ES.2 indicates, 
the program group was slightly more likely (98 percent versus 93 percent) than the control group 
to have been enrolled in an educational program. Almost two-thirds of girls in the program group 


Centers for Disease Control and Prevention, “About Adverse Childhood Experiences” (2016), 
https://www.cdc.gov/violenceprevention/acestudy/about_ace.html. 

‘Quantitative data sources include PACE’s management information system, administrative records from 
the Florida Department of Juvenile Justice and the Florida Department of Education, and a follow-up survey 
fielded to the full study sample 12 months after study enrollment. Qualitative data include in-depth, follow-up 
phone interviews with a nonrandom subset of study participants and parents. The cost analysis draws on infor- 
mation from PACE’s central management office about revenue and expenditures at both the center and organi- 
zation levels. 
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reported received counseling or therapy, compared with 43 percent of those in the control group, 
and they received the service more frequently (not shown). 


Figure ES.2 also shows that the program group was more likely than the control group to 
have received these services from a professional source (usually PACE). In the absence of PACE, 
the control group may have had to piece together academic and social services from different 
sources. A service provided by a staff member at an organization or school is probably closer in 
content to the services received by girls at PACE than a service that is provided by a parent, family 
member, or friend. 


Figure ES.2 
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NOTES: Results in this figure are regression-adjusted, controlling for pre-random assignment characteristics. 
Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 
Rounding may cause slight discrepancies in sums and differences. 


Girls in the control group were usually enrolled in school and moderately engaged in 
receiving other academic and social services.'* Overall, however, a smaller portion of the control 
group received services that included key aspects of PACE’s model, such as family engagement 
and supportive relationships between staff members and girls. 


e Over a one-year period, PACE increased school enrollment and attend- 
ance for the girls it served, compared with the control group. Girls in the 
program group were also more likely to be “on track” academically than 
those in the control group. 


As shown in Table ES.1, program group members were, on average, present in school 
for about 10 more days than control group members over the full calendar year following study 
enrollment — two full weeks of additional academic instruction.!? The impact appears to be due 
to girls in the program group being enrolled for more days and less likely to be absent throughout 
the year — particularly in the summer term, where there was a substantial impact (27 percentage 
points), which reflects PACE’s year-round program structure. Some research suggests that being 
out of school during the summer results in learning loss. This loss is usually larger for lower- 
income students, like many girls at PACE, than for higher-income students, who may have access 
to additional learning resources over the summer. 4 


Table ES.1 also presents information on academic progress. Many girls who applied to 
PACE needed help getting back on track academically. These findings indicate that PACE’s suite 
of services helped them do just that: High school girls in the program group were more likely (by 
13 percentage points) than the control group to be on track academically, as measured by a com- 
posite that considers a student on track if she has a high attendance rate, has not been expelled or 
suspended, and has not failed a core course. !° These components reflect what researchers consider 
to be predictors of high school graduation — attendance, behavior, and course performance — 
often referred to in the field as the “ABCs.”!* Girls across the two groups earned a similar number 
of credits during the academic year. 


?Florida law requires all children under the age of 16 to be enrolled in school. 

>The full calendar year includes both the academic year and the summer term. The academic year refers to 
the traditional school term in Florida, which runs from about mid-August through the end of May. 

‘David M. Quinn and Morgan Polikoff, “Summer Learning Loss: What Is It, and What Can We Do About 
It?” (Washington, DC: Brookings Institution, 2017), https://www.brookings.edu/research/summer-learning 
-loss-what-is-it-and-what-can-we-do-about-it/. 

Core courses include English language arts, math, science, and social studies. Being absent for less than 
10 percent of days enrolled is considered high attendance. The composite measure and credits earned are avail- 
able for high school students only. 

®Elaine M. Allensworth, Jenny Nagaoka, and David W. Johnson, High School Graduation and College 
Readiness Indicator Systems: What We Know, What We Need to Know (Chicago: University of Chicago Con- 
sortium on School Research, 2018). There is no single set of thresholds across these predictors that the field 
widely uses to measure the likelihood of high school graduation. 
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Table ES.1 


Impacts on Key Outcomes 
During the 12- to 18-Month Follow-Up Period 


Program Control Difference 
Outcome Group Group __ (Impact) P-Value 


Academic engagement and progress* 


Number of days present 119.6 109.8 9.8 ** 0.013 
Ever enrolled in summer term (%) 39.4 12.6 26.8 *** 0.000 
On track, based on composite measure”® (%) 27.6 14.2 13.4 *™* 0.000 
Credits earned in academic year® 3.8 3.7 0.1 0.696 


Youth development and risky behavior® 


Has supportive adult in her life® (%) 60.9 61.1 -0.2 0.945 
Sexually active and did not use pregnancy protection 

method during last sexual encounter’ 8.8 8.6 0.3 0.889 
Incurred a charge since random assignment? 22.0 21.2 0.8 0.739 
Thinks she will meet or exceed education goals 74.9 72.8 2.0 0.510 
Sample size 673 452 


SOURCE: MDRC calculations based on data from the Florida Department of Education (FL DOE), responses to 
the PACE evaluation 12-month follow-up survey, and data from the Florida Department of Juvenile Justice (FL 
DJJ). 


NOTES: Results in this table are regression-adjusted, controlling for pre-random assignment characteristics. 
Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Rounding may cause slight discrepancies in sums and differences. 

4aOutcomes draw on data from FL DOE and refer only to involvement in the Florida school system. The 
measures cover the full calendar year, which includes the academic and summer term, except where noted. 

>The composite includes the following criteria: absent less than 10 percent of days; never failed a core course; 
never expelled or suspended. Core classes include English language arts, math, science, and social studies. 

eSample includes high school students only. 

dOutcomes are from self-reported survey data, unless otherwise indicated. 

®Other than parent or guardian. 

‘Measure includes only voluntary sexual encounters. 

JIn the juvenile justice system, people are not technically "arrested"; the terminology used is either "incurred a 
charge" or "referred." This measure uses the FL DJJ data and covers involvement in the 18-month period following 
random assignment. 


The differences in academic outcomes are promising. Engagement in school is a protec- 
tive factor against involvement in the justice system. Longer-term follow-up on the study sample 
would provide answers about whether these short-term differences lead to higher high school 
graduation rates or more students graduating on time. 
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e Sample members across both the program and control groups appeared 
to be goal-oriented and hopeful about their futures and reported rela- 
tively low levels of risky behavior one year after study enrollment. Rates 
of formal involvement in the juvenile justice system were similar for the 
program and control groups. 


There were a few differences between groups on self-reported measures of youth devel- 
opment and risky behavior, but no clear pattern or trend. The positive outlook expressed by sam- 
ple members across both groups stands in contrast to the crisis state or tipping point that many 
sample members were experiencing at the time they applied to PACE. One hypothesis is that the 
girls’ lives stabilized somewhat as the crisis passed or as girls found resources to address their 
needs. Levels were relatively low for risky behaviors involving substance use, similar to those of 
the broader population of girls in Florida and nationwide. '” 


Sample members’ involvement in the juvenile justice system was measured across several 
outcomes, including whether they ever incurred a charge and the type of adjudication finding after 
a charge was incurred. Incurring a charge is comparable to an arrest in the adult criminal justice 
system, and being “adjudicated delinquent’ is similar to a conviction. As shown in Table ES.1, 
program and control group girls incurred one or more charges at nearly identical rates (22 percent 
of the program group and 21 percent of the control group) during an 18-month follow-up period. 


The rates of justice system involvement for both groups are higher than that of the broader 
population, '* reflecting the segment of girls PACE aims to serve. Yet sample members were just 
reaching the age where criminal behavior starts to emerge,!? so it may be too early to expect 
impacts on these measures. Further follow-up with the study sample would be necessary to see 
whether differences emerge between the two groups. 


Notably, practices in the juvenile justice system are shifting. Recently, there has been a 
nationwide movement to confine fewer delinquent youth.”° In Florida, arrests of young people 
overall have fallen dramatically in the last few years, and females are making up a smaller 


Laura Kann, Tim McManus, William A. Harris, Shari L. Shanklin, Katherine H. Flint, Joseph 
Hawkins, Barbara Queen, Richard Lowry, Emily O’Malley Olsen, David Chyen, Lisa Whittle, Jemekia 
Thornton, Connie Lim, Yoshimi Yamakawa, Nancy Brener, and Stephanie Zaza, “Youth Risk Behavior 
Surveillance — United States, 2015,” Morbidity and Mortality Weekly Report: Surveillance Summaries 
65, no. 6 (2016): 1-174. 

'8Across the state of Florida, approximately 1 percent of girls were arrested in 2014. 

'SNational Institute of Justice, “From Juvenile Delinquency to Young Adult Offending” (2014), 
https://www.nij.gov/topics/crime/Pages/delinquency-to-adult-offending.aspx. 

°Child Trends Data Bank, Juvenile Detention: Indicators on Children and Youth (2015), 
https://www.childtrends.org/wp-content/uploads/2012/05/88 Juvenile Detention.pdf. In 2006, 289 per 100,000 
young people resided in a juvenile detention center, correctional facility, or residential facility. By 2013, the rate 
had dropped to 173 per 100,000 young people. 
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percentage of juvenile arrests.*! PACE works closely with the Florida Department of Juvenile 
Justice, the state legislature, and other entities to advocate for these types of policy shifts. 


e The cost of PACE’s holistic package of services is, on average, $10,400 
more than the cost of the services received by the control group. The anal- 
ysis finds that the cost of providing academic services at PACE is compa- 
rable to the cost of public school; PACE’s wrap-around services account 
for the difference in costs. 


During the years the program group members were receiving PACE services, the PACE 
program cost about $23,500 per girl served. (All costs are expressed in 2017 dollars.) This figure 
is based on the average length of stay for girls in the study (eight months). When compared with 
the estimated cost of services used by the control group during the same period, the net cost of 
the program group services is about $10,400. Though academic costs were similar overall for the 
two groups, social services (provided mainly by PACE) accounted for most of the differences in 
costs. Such intensive social service support is rare in the public schools. PACE serves a population 
that, on average, needs more services than traditional schools provide, so its costs are unsurpris- 
ingly higher. 


While there are few similar programs for at-risk children, it is useful to consider PACE’s 
costs in the spectrum of services for young people: PACE is less costly than a residential program, 
similar in cost to other comprehensive youth programs, and more expensive than public schools. 
The small size of PACE centers (serving about 50 girls, on average) and a staffing ratio much 
lower than that typically found in traditional schools contribute to its costs. 


Ultimately, the aim of PACE is to change the long-term trajectory of its participants by 
getting them back on track academically and promoting more prosocial behavior. If this occurs, 
the benefits PACE produces would accrue over a much longer time horizon than the eight-month 
period when society is making its investment in the girls at PACE. For example, if PACE were 
able to improve girls’ graduation rates by at least 8 percentage points, the program would pay for 
itself in the long term. 


PACE’s Ongoing Improvements 


This evaluation and the promising academic outcomes reflect the services girls received during 
the evaluation period, from 2013 to 2015. PACE, which has a strong central office to support 
program implementation, has opened more centers across the state since the evaluation began, 
and PACE’s leaders continue to assess their services and how they are implemented. Over the 
course of the evaluation period, PACE instituted a new management structure at its central office, 
revised policies on program eligibility, and began revamping the life skills curriculum, among 
other changes made. 


*!Florida Department of Juvenile Justice, “Delinquency Profile 2017: Statewide Intake — Arrests” (2017), 
http://www.djj.state.fl.us. In fiscal year 2015-2016, 38,267 young people (ages 10 to 17) were arrested in Florida, 
down from 57,597 arrests in fiscal year 2011-2012. 
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The leadership team at PACE also worked actively with the study team to understand 
and address specific findings from the implementation research. For example, the study team 
found that there was no standard set of approaches recommended for counselors to use in sessions 
with girls; as a result, PACE developed a more defined toolbox for counseling. PACE has also 
strengthened teacher training on instructional methods and shifted away from independent work 
in the classroom. In addition, PACE rolled out new in-person staff training on its key pillars as a 
gender-responsive, strengths-based, and trauma-informed program — terminology that staff 
members did not necessarily recognize, even as they uniformly implemented such practices. 


Other initiatives are under way as well. As one executive director noted recently, there is 
an increase in support for service delivery and a move to standardize delivery while allowing for 
local, center-level differences. In addition, PACE is focusing on enhancing its use of the data it 
collects, including implementing a new management information system. 


Implications of the Evaluation 


The evaluation concludes that PACE employs a unique, well-implemented, theory-based model, 
as discussed at length in the previous report.”” While girls in the control group did receive other 
services, the model’s gender-responsive approach and combination of academic and social ser- 
vices are not commonly found. 


Based on the random assignment study design, the evaluation also concludes that the 
PACE program had a positive effect on girls at risk of juvenile justice involvement in the short 
term, leading them to be more engaged and on track academically. Girls in both research groups 
seemed to have stabilized a year after applying to PACE, and effects on other outcomes, including 
risky behavior and youth development, were not seen in the short term, 12 to 18 months after 
enrollment. Overall, the girls in the study sample had positive relationships with friends, family, 
and adults outside their families, and rates of substance use were no higher than those of the 
broader population of girls. 


These encouraging outcomes may in part be due to PACE’s eligibility requirements: 
Girls had to have a parent or guardian present during the intake assessment and to show a will- 
ingness to change things in their lives. This combination applied equally to all girls in the study. 
The girls in the control group, who were referred to other services by PACE, also may have been 
motivated and supported in finding services they needed in the absence of access to PACE. 


The services PACE offers girls are more expensive than the less holistic alternatives 
available in the girls’ communities. The study’s short follow-up period does not permit a full cost- 
benefit analysis that would indicate how this investment pays off in the long run. It is plausible 
that increased engagement and being on track academically would lead to both higher graduation 
rates and lower justice system involvement; a substantial increase in the graduation rate alone 
would make the program cost effective. It will be four to five years, however, before the vast 


?Treskon, Millenky, and Freedman, Helping Girls Get Back on Track. 
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majority of the study sample reaches the timing for high school graduation. Possible differences 
on juvenile justice outcomes may emerge before then. 


The evaluation of the PACE Center for Girls adds to the growing literature regarding 
programs serving girls at risk of justice system involvement and other unfavorable outcomes. It 
also sheds new light on the implementation of gender-responsive services tailored to girls’ com- 
mon experiences and individual strengths. The evaluation finds that PACE reengages girls in 
academics in a way that could make a lasting difference, especially if it leads to higher rates of 
high school graduation and enrollment in postsecondary education. 
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Chapter 1 


Introduction 


PACE Center for Girls is a Florida-based nonprofit organization that targets girls of middle school 
and high school age who are at risk of involvement with the justice system. These girls are often 
struggling academically; some have histories of juvenile justice involvement; some have mental 
health issues, often stemming from experiences of trauma; and some engage in behavior that 
negatively affects their physical health. Using a “‘gender-responsive” framework for its academic 
and social services, PACE seeks to reduce the likelihood of negative outcomes tied to this set of 
characteristics and backgrounds and instead foster academic engagement, positive youth devel- 
opment, and healthy relationships. Increasing these protective factors can improve girls’ aca- 
demic outcomes, prevent their future involvement with the justice system, and lead to long-term 
well-being and success. This report presents findings from a rigorous evaluation of PACE. 


Improving academic outcomes in the short term may help girls get back on track and 
graduate on time. Preventing involvement in the juvenile justice system is also crucial, for several 
reasons. Juvenile justice involvement can damage a young person’s relationships with friends and 
family, negatively affect mental health, and interrupt the academic progress and work experience 
that should accumulate during adolescence. ! 


Investing in the well-being of girls promises economic benefits as well. Compared with 
dropouts, high school graduates have been shown to make larger economic contributions to soci- 
ety through lower unemployment rates, higher earnings, more taxes paid, and less reliance on 
public assistance. One estimate found that high school graduates make nearly $300,000 more than 
those who do not complete high school over the course of their lifetimes.” And from a societal 
perspective, the court and detainment costs associated with juvenile justice involvement are high. 
Therefore, effective prevention or early intervention programs that help young people avoid in- 
volvement with the juvenile justice system can offer a significant return on investment.* 


PACE uses principles of gender-responsive programming throughout its services for 
girls. The term “gender-responsive” describes treatment approaches designed for girls and 
women, specifically those involved with or at risk of involvement with the juvenile and criminal 
justice systems. Historically, services for those involved with the justice system were designed 
for boys and men.* Gender-responsive services bring an awareness of girls’ distinctive develop- 
ment patterns and gender-specific issues into the program. 


‘Aizer and Doyle (2015). 


?This calculation compares lifetime earnings for high school dropouts with earnings for high school gradu- 
ates with no further education (Tamborini, Kim, and Sakamoto 2015). 


3Aos et al. (2004). 
4Treskon, Millenky, and Freedman (2017); Zahn, Hawkins, Chiancone, and Whitworth (2008). 


Some girls’ risk factors for delinquency are similar to those of boys, but they may mani- 
fest themselves differently in girls. Girls involved in the juvenile justice system are more likely 
than boys to have a history of maltreatment and other trauma, running away, family conflicts, 
exposure to crime in the neighborhood or at school, chronic mental and physical health disorders, 
substance abuse, and academic disruptions. Their experience of abuse is striking, with reported 
prevalence as high as 92 percent in one study of female delinquents.° Some risk factors, such as 
dating much older partners and self-harm, are almost never seen in male offenders. 


Although there is some variability, many descriptions of gender-responsive program- 
ming are markedly consistent. The Center for Gender and Justice defines gender-responsive 
services as those that “creat[e] an environment through site selection, staff selection, program 
development, content, and material that reflects an understanding of the realities of the lives of 
women and girls and that addresses and responds to their strengths and challenges.”® Among 
the common principles are special attention to relationships, including those between staff 
members and girls; an emphasis on safety; an awareness of and response to sexism; a strengths- 
based approach, which builds on girls’ individual strengths rather than focusing on their defi- 
cits; a holistic approach, addressing the well-being of the whole person rather than focusing on 
problems individually; and family involvement, including the development of positive family 
connections. Services often included in a gender-responsive program are treatment for abuse 
and trauma, life skills education, educational and vocational opportunities, and community and 
volunteer experiences.’ 


These types of services have been promoted at the state and national levels.* Yet the cur- 
rent literature on gender-responsive programming is more robust in its description of concepts 
and principles than in its evaluation of program performance.’ Until recently, it was largely un- 
known how gender-responsive services are implemented, how similar they are to one another, or 
how effective they are.'° Recent research suggests that, as with many other social services, 
gender-responsive program models may not always be implemented as planned, and the pro- 
grams themselves are believed to be insufficient in number and not necessarily targeting greatest 


Treskon and Bright (2017). 

®Covington and Bloom (2017). 

7See Treskon and Bright (2017) for more information on the gender-responsive approach and Treskon, 
Millenky, and Freedman (2017) for how this approach translates to a real-world setting at PACE Center for Girls. 


SAt the federal level, support for a gender-responsive approach crystallized with the 1992 amendment to the 
Juvenile Justice Delinquency Prevention Act, which remains in effect to this day (Juvenile Justice and Delin- 
quency Prevention Act, 42 U.S.C. 5633 § 242 [1992]). Currently, the U.S. Department of Justice’s Office of 
Juvenile Justice and Delinquency Prevention partners with a national organization on the National Girls Initia- 
tive, which provides training, technical assistance, and other resources to programs serving this population 
(Treskon and Bright 2017). 

°Hubbard and Matthews (2008). 


‘OChesney-Lind, Morash, and Stevens (2008); Kerig and Schindler (2013). 


need.!! A clearer picture of program operations, populations served, and outcomes would aid in 
determining how to implement gender-responsive services effectively. 


This report describes the results of a rigorous evaluation of PACE that began in 2013.!° 
The evaluation was an opportunity to test the effectiveness of the program and provide evidence 
on gender-responsive programming. The evaluation was conducted by MDRC and has been 
funded mainly through the Edna McConnell Clark Foundation’s Social Innovation Fund (SIF), a 
program of the Corporation for National and Community Service (CNCS), with additional fund- 
ing provided by the Jessie Ball duPont Fund and the Healy Foundation. 


This final report focuses mainly on measuring the extent to which PACE positively af- 
fects the girls that it serves in the short term. In brief, the program group received more services 
and received them more often from a professional source than the control group did. The evalua- 
tion found that PACE had promising academic outcomes for the girls it served, compared with 
the control group. It did not affect rates of risky behavior, including formal juvenile justice in- 
volvement, over the follow-up period. The average program cost per girl was $23,499; compared 
with the cost of the services used by the comparison group, the net cost was approximately 
$10,400. Given the age of the girls PACE serves, it is too soon to tell how PACE may affect girls’ 
longer-term outcomes in the areas of education and delinquency, or in other areas, and whether it 
is cost effective in the long run. 


PACE Center for Girls 


PACE Center for Girls, established in 1985, currently operates 21 nonresidential program loca- 
tions (known as “centers’”’) throughout the state of Florida. PACE is a voluntary program that 
serves girls ages 11 to 18 whose experiences, often shortly before coming to PACE, include abuse 
or violence, poor academic performance, or truancy — risk factors for delinquency. Some PACE 
applicants have already been involved with the juvenile justice system, and all are at risk of future 
involvement. Girls and their parents or guardians often come to PACE through referrals by 
the school district or a local organization; others learn about PACE through word of mouth. 


Before enrollment, PACE applicants must undergo a thorough application process to 
determine eligibility. Parents or guardians are also involved during this process, as they must 
be physically present and agree to the girl’s participation in PACE. One of the initial require- 
ments for program eligibility is having risk factors in multiple domains; the likelihood of 
involvement with the justice system increases with the presence of multiple risk factors. 4 
Other major considerations are PACE’s ability to serve a girl’s academic and social service 


'\Chesney-Lind, Morash, and Stevens (2008). 
Treskon and Bright (2017). 


'3Previous publications from the evaluation introduced the project and the PACE program model, presented 
findings on how PACE implements that model, and focused specifically on the gender-responsive programming. 
See Millenky and Mage (2016), Treskon, Millenky, and Freedman (2017), and Treskon and Bright (2017). 


'4Wasserman et al. (2003). 


needs and the girl’s motivation to attend the program and acknowledgment that change is 
needed in her life. 


As shown in Figure 1.1, PACE provides daily academic services along with a significant 
set of social services; as discussed above, the program uses a gender-responsive approach. Girls 
attend PACE during normal school hours and stay enrolled for about a year. Enrollments in PACE 
occur throughout the full calendar year, including over the summer. Most girls then make the 
transition to another school in the community to complete their education. Through these services, 
PACE aims to increase protective factors for the girls it serves. More specifically, the PACE 
program model includes the following components: 


e Academic instruction and advising. Girls receive daily instruction at the 
middle school or high school level in a small class setting. Individual academic 
plans guide progress, which is monitored through biweekly advising sessions. 


e Individual assessment and care planning. Assessments of each girl’s needs 
are used to create tailored plans for the girl’s time at PACE. Staff members 
meet regularly to share information and review progress. Staff members refer 
girls to services outside PACE as needed. 


e Individual and group counseling. Girls attend individual sessions with coun- 
selors at least every other week and have access to psychoeducational group ses- 
sions. The individual sessions are guided by a review of the girl’s care plan goals. 
Examples of goals are building healthy relationships, decreasing substance 
abuse, improving anger management skills, and increasing positive self-esteem. 


e Life skills curriculum. Girls regularly attend a Spirited Girls! class that covers 
six domains believed to be essential for girls’ healthy development: physical, 
emotional, intellectual, relational, sexual, and spiritual domains. 


e Parental engagement. Program staff members work to engage parents 
through initial home visits, monthly progress reports, office sessions, and 
phone contact. 


e Volunteer service and career exploration. Girls engage in volunteer oppor- 
tunities in their communities; career exploration activities are often offered as 
well. 


e Transition and follow-up services. Additional support is available for girls 
as they make the transition out of PACE and back to their home schools or 
other appropriate placements. Staff members also check in with girls at regular 
intervals for one year after they leave the program to provide services or refer- 
rals, if needed. 


Inputs 


Experienced and 

trained academic 

and social service 
staff 


Safe environment 
that is responsive to 
the needs of girls 


Central office 
that oversees local 
centers 


Financial support 
from public and 
private sources 


Various community 
partners 


Participant characteristics 
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PACE Center for Girls Logic Model 


Program Components 


Academic Services 
Academic instruction and advising 
Life skills curriculum (Spirited Girls!) 


Volunteer service and career exploration 


Social Services 
Individual assessment and care planning 
Individual and group counseling 
Parental engagement 


Transition and follow-up services 


Center characteristics 


Outcomes 


Short-Term 


Academic 
engagement and 
progress 


Reduced risky 
behavior 


Improved 
interpersonal skills 


Improved self- 
advocacy and 
self-efficacy 


Preparation for 
future 


Long-Term 


Girls complete high school 
or receive equivalency 
diploma. 


Girls are not involved with 
the criminal justice system. 


Girls have appropriate and 
healthy behaviors and 
relationships. 


Girls are prepared to 
advocate for themselves 
and their communities. 


Girls have strong coping 
and decision-making skills 
and know how to get help 
when necessary. 


Girls are in college, 
training, or a job. Girls are 
economically stable. 


Community context 


As shown in Figure 1.1, participation by girls in this suite of services — designed to work 
holistically — is expected to lead to some positive short-term outcomes. Academically, these 
short-term outcomes include improved engagement, such as increased attendance at school, and 
progress as measured by grade promotion and earning credits. In the longer term, this engagement 
and progress could result in more girls graduating from high school or earning a high school 
equivalency credential. PACE may also help girls in the short term reduce their risky behavior, 
such as delinquency or high-risk sexual activity, and improve their interpersonal skills, their self- 
efficacy, and their knowledge of career options. 


In addition to high school completion, involvement in PACE may lead to other positive, 
longer-term outcomes for participants. As noted in Figure 1.1, these include lower rates of in- 
volvement with the criminal justice system, better coping and advocacy skills, healthier relation- 
ships and behaviors, and a positive transition to adulthood. This transition to adulthood may be 
reflected in participation in postsecondary education, job training, or employment. 


PACE is primarily government funded, receiving more than two-thirds of its funding 
through the state’s educational system and the Florida Department of Juvenile Justice (DJJ). For 
academic services, the centers contract with the local school districts to receive per pupil funding. 
The funding from DJJ supports a large part of the social services as well as facility expenses. 
Additional support comes from federal, state, and local grants as well as donations from individ- 
uals, corporations, and foundations. 


PACE Evaluation 


The evaluation of PACE’s program model presents an opportunity to test the effectiveness of a 
gender-responsive program and provide rigorous evidence on services for girls at risk of negative 
outcomes. The evaluation had three main components: an implementation study, an impact study, 
and a cost analysis. Fourteen PACE centers participated in the evaluation. 


The implementation study assessed how well PACE implements its program model and 
incorporates gender-responsive principles into services. Findings from the implementation study 
were published in a 2017 interim report titled Helping Girls Get Back on Track.'> The report also 
included information on the recruitment and application process for the evaluation and prelimi- 
nary impacts on service receipt. A research brief that focused on gender-responsive programming 
served as a companion to that report and used PACE as a case study of how the principles of that 
approach are put into action.!° 


The key findings of the implementation study included the following: 


e The PACE program model was consistently implemented across multiple lo- 
cations, with a core set of similar services. 


'STreskon, Millenky, and Freedman (2017). 
‘6Treskon and Bright (2017). 


e PACE incorporated gender-responsive programming into its services by cre- 
ating a safe, relationship-focused environment and by using tools and ap- 
proaches that fit within the gender-responsive principles. 


e Although the content of academic classes was similar to what girls would 
learn in regular public schools, academics at PACE differed in student- 
teacher ratio, availability of academic advising, and a focus on individual, 
self-paced work. Classes were provided in the context of a gender-responsive 
program environment. 


The current report focuses mainly on the impact and cost analyses. The impact analysis 
measures effects on short-term outcomes, including academic engagement and progress, juvenile 
justice involvement, risky behaviors, and interpersonal relationships. An extended follow-up pe- 
riod would be necessary to measure the longer-term outcomes discussed earlier, given the age 
range of the girls served by PACE. The impact findings are placed in context through information 
about the services both groups participated in during this follow-up period. The cost analysis 
presents the costs of providing PACE services and compares these costs with those of alternative 
services. Since a year is too short a time frame to expect to fully recoup the costs of such an 
intensive program, the report then discusses the types of longer-term impacts that PACE would 
need to have in order to be cost neutral. 


The impact analysis employs a random assignment design: Girls who applied to and were 
deemed eligible for PACE (using the program’s existing screening processes) enrolled in the 
study and were assigned at random either to a program group, whose members were offered 
PACE services, or to a control group, whose members received appropriate referrals to other 
services in the community. Figure 1.2 shows the place of random assignment in relation to the 
other steps in the eligibility and enrollment process. The “intent-to-treat” design employed in the 
impact analyses shown throughout the report includes a// girls assigned to each group in the cal- 
culations, regardless of their participation in PACE or other services. 


Over a period of more than two years (from August 2013 to November 2015), 1,125 girls 
enrolled in the study (673 in the program group, 452 in the control group). Random assignment 
was stratified at the center level; among each center’s study enrollees, 60 percent were assigned 
to the program group and 40 percent to the control group. Some girls, such as those in foster care 
or those who had previously attended PACE, were not eligible for the study but were still invited 
to enroll in PACE. (See Appendix A for further information about site selection and random as- 
signment procedures.) Data were collected on both program and control group outcomes for least 
12 months following random assignment, to measure any differences that emerged across the two 
groups. 


Figure 1.2 


PACE Evaluation Eligibility and Enrollment Process 


Referral 
PACE receives referrals for girls from partners such as the school district, 
Department of Juvenile Justice, and community-based organizations, and 
directly from the girls’ families. 


y 


Introduction and Intake 
Girls are accompanied by a parent or guardian for the intake appointment 
at the PACE center. 


Introduction to PACE Orientation to PACE Paperwork and 
evaluation assessment 

Girls and their families Girls and their families A staff person conducts 
learn about services receive information a formal assessment 

offered by PACE, meet about the study and, if with each girl, covering 

staff members, and tour they agree to participate, topics such as her 

the center. give consent. background and risk 
factors. 


y 


Eligibility Determination 


A group of staff members at the center discuss each girl’s eligibility for PACE and 
whether PACE is appropriate for the girl. 


} 


Random Assignment 


Girls determined to be eligible for PACE are randomly assigned. 


Invitation to join PACE Referral to other resources 
Girls randomly assigned to the Girls assigned to the comparison 
program group are invited to enroll group are provided with information 
at PACE as soon as a space or referrals to other services 

is available. in the community. 


Data Sources 


The analyses in this report draw on the following data sources: 


PACE management information system (MIS). PACE centers collect ex- 
tensive data about girls in a centrally managed MIS. These data include base- 
line demographic and risk factor information — collected during the program 
application process and before study enrollment — and extensive records on 
participation in program services. 


12-month follow-up survey. A follow-up survey was administered to study 
participants by phone or in person approximately one year after study enroll- 
ment. The information collected in this survey is used to measure impacts on 
service receipt, academic progress and engagement, risky behavior, and youth 
development, as well as to understand specific details about services and help 
received by the control group. The response rate among program group mem- 
bers was 79 percent, slightly higher than the rate among the control group (76 
percent). Appendix B includes more details on fielding of the survey and find- 
ings from a survey response bias analysis. 


Administrative records from the Florida Department of Juvenile Justice. 
The research team obtained records from the Florida Department of Juvenile 
Justice to measure impacts on study participants’ involvement in the juvenile 
justice system. These data include information from 2008 to 2017, allowing 
the research team to estimate impacts for the full study sample at 18 months 
after study enrollment and measure any involvement before study enrollment. 


Administrative records from the Florida Department of Education. Using 
records from the Florida Department of Education, the research team meas- 
ured impacts on academic progress, academic engagement, and disciplinary 
actions. The data come from the 14 (county-level) school districts where 
PACE centers are located and cover the 2013-2014, 2014-2015, and 2015- 
2016 school years, allowing the research team to estimate impacts for a one- 
year period for the entire study sample. 


Program cost data. PACE headquarters provided data about each participat- 
ing center’s revenue and expenditures in fiscal year 2015 (July 2014 to June 
2015). Financial information for the headquarters is included as well. 


Semistructured phone interviews conducted with sample members and 
parents. Girls who completed the 12-month follow-up survey and their par- 
ents or guardians were asked to participate in an additional in-depth, semi- 
structured interview about their experiences with PACE and other schooling 
or services. A subset of those who expressed interest were contacted. A total 


of 52 girls and 40 parents who agreed to participate were contacted and inter- 
viewed by phone between July 2015 and March 2016. See Appendix D for 
more details. 


While the evaluation uses a rigorous design and draws on a range of data sources, it has 
some limitations. The evaluation’s funding allowed for data collection for only 12 to 18 months 
following the girls’ application to PACE and enrollment in the study. While this time period 
provides information on PACE’s short-term outcomes, it was not enough time for most girls in 
the study sample to reach key milestones such as high school graduation. Therefore, the evalu- 
ation does not include a full benefit-cost analysis at this time. Additionally, aspects of youth 
development can be hard to measure. As is the case in many studies of youth programs, the 
follow-up survey asked sample members questions that assess risky behavior and interpersonal 
relationships; yet these constructs remain difficult to measure, especially in a short survey that 
was usually administered by phone. Among other things, girls may be hesitant to provide honest 
or complete answers to a survey interviewer about ongoing issues or risky behaviors. Finally, 
this study benefited from access to Florida’s Department of Education records, but the study 
team had access only to deidentified data, which limited the analyses that could be performed. 


Characteristics of the Study Sample 


Figures 1.3, 1.4, and 1.5 present the study sample members’ demographic characteristics 
and risk factors. This information was collected from girls and parents by PACE program staff 
members at the time of their application to PACE and study enrollment. Most girls were between 
13 and 16 at the time of study enrollment; the average age across the whole sample is 14.7. Just 
over half the sample came from single-parent households. Appendix Tables C.1 and C.2 include 
additional measures and a breakdown of the study sample by research group. Overall, there are 
very few differences between the two research groups. 


Consistent with the characteristics of PACE’s target population, a range of risk factors 
were prevalent across the study sample. Many sample members struggled with school before 
coming to PACE, as shown in Figure 1.4. At the time of study enrollment, 40 percent had been 
recently expelled or suspended from school, and just over half the sample had been held back at 
least once. These rates are considerably higher than those among all students in the counties where 
PACE centers are located; in the 2013-2014 school year, only 15 percent of all students were 
expelled or suspended, and just 5 percent were held back.!” 


'’These measures were calculated from Florida Department of Education data for the counties where par- 
ticipating PACE centers are located. The “held back” measure represents the rate for students in grades 6 through 
12. Expulsion and suspension data were not available by grade level. 
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Figure 1.3 


Sample Members’ Characteristics at Study Enrollment 
Age 
11-12 
13-14 
15-16 
17 and up 


Ma s% 

es ;: 
ee 
MM 10% 


Race/ethnicity* 


es 45 
White, non-Hispanic [i 33% 
Hispanic 16 % 


Other | 1% 


Black, non-Hispanic 


Grade 


5th-6th grade 
7th-8th grade 
9th-10th grade 
11th-12th grade 


| 
CS 
eC SE 
Ma 9% 

Learning disability 


None 


en 0% 
a 20". 
Dyslexia ff 1% 


Other SJ 9% 


People participant lives with 


Single parent [iS 52° 
Two parents iS 35% 
Relative QR 11% 


Other J 3% 


Attention deficit disorder 


Family income 


$28,050 or below [iS 42% 
$28,051-$44,900 3s °, 
Above $44,900 x 23% 


SOURCE: MDRC calculations based on data from the PACE management information system. 
NOTES: Sample size is 1,125. 


aSample members are coded as Hispanic if they answered "yes" to Hispanic ethnicity. 
"Other" includes nonrelative or foster care. 
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Figure 1.4 


Sample Members’ Academic Risk Factors at Study Enrollment 


Recently expelled or 


Lo} 
opened! as 
Not currently enrolled in 
Had more than 15 absences e 
; 35% 
in past 3 months 


Held back at least once 52% 


Failed 1 or more classes in 
past 6 months 


77% 


SOURCE: MDRC calculations based on data from the PACE management information system. 


NOTES: Sample size is 1,125. 

@This measure was captured in two different ways during the random assignment period. It was defined as 
being currently expelled or suspended for approximately half of the sample and as one or more expulsions or 
suspensions in the most recent school term for the other half. 


As Figure 1.5 indicates, risk factors for delinquency were also very common among sam- 
ple members. Just over a quarter of the study sample reported a prior arrest.'* Unsurprisingly, 
given PACE’s target population, this is far higher than the arrest rate for the broader Florida fe- 
male youth population, which was only about | percent in 2014.'° Additionally, sample members 
often had people in their lives with a criminal or delinquent history. Almost two-thirds of the 
research sample had a family member with a criminal history. A growing body of research indi- 
cates that children with incarcerated parents often experience trauma, family disruption, and the 
loss of their primary caregiver, which can lead to financial hardship, residential instability, and 
an array of emotional and behavioral problems.*° Furthermore, half of the sample reportedly had 
friends with prior juvenile justice involvement or who engaged in delinquent behavior. 


Girls in the study also reported many health and safety risk factors, such as sexual activ- 
ity, a history of abuse, and substance use. At the time of random assignment, almost half the 
sample members (44 percent) had ever been sexually active. Thirty-eight percent of the research 
sample reported having been abused or neglected.?! Smaller portions of the sample were using 


'8Technically, these are not arrests; in the juvenile justice system, the corresponding term for an arrest is 
“incur a charge.” This statistic from PACE data aligns with the girls’ prior rate of incurring charges as calculated 
from state-level administrative records. 

'Florida Department of Juvenile Justice (2017). 

0Parke and Clarke-Stewart (2001). 

21Similarly, 43 percent of sample members had a family case with the Florida Department of Children and 
Families, which handles abuse and neglect cases. 
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Figure 1.5 


Sample Members’ Delinquency and Health and Safety Risk Factors 
at Study Enrollment 


Delinquency 


Ever incurred a charge? 28% 


Currently on probation 13% 
Has family member with 


criminal history® 64% 


Has friends with delinquent 
record or who engage in 
delinquent behavior 


50% 


Health and safety 


Currently using tobacco* 10% 


Currently using drugs or 
alcohol¢ 


15% 


Ever sexually active 44% 


Ever run away from home 28% 
Ever had thoughts about 


Lo} 
harming or killing herself aa 


Abused or neglected® 38% 
Family has case with the 
Florida Department of 
Children and Families 


43% 


SOURCE: MDRC calculations based on data from the PACE management information system. 


NOTES: Sample size is 1,125. Certain characteristics listed here were captured in two different ways during 
the random assignment period, as noted below. 

aln the juvenile justice system, people are not technically "arrested"; the terminology used is either "incurred 
a charge" or "referred." 

>For approximately half of the sample, this refers to a criminal record (including imprisonment, probation, 
parole, and house arrest) for a parent, guardian, or sibling of the sample member; for the other half of the 
sample, "family" includes other members of the household as well. 

cMeasure was defined as having used tobacco three or more times in the past 30 days for approximately 
half of the sample and defined as currently using tobacco for the other half. 

dMeasure was defined as having used drugs or alcohol three or more times in past 30 days for 
approximately half of the sample and defined as current drug or alcohol use for the other half. 

*Measure refers only to documented instances of abuse or neglect for approximately half of the sample but 
includes suspected incidents of abuse for the other half. 
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drugs, alcohol, or tobacco at the time study enrollment: 15 percent of sample members were using 
drugs or alcohol, and 10 percent were using tobacco. 


To put this into context, the rate of sexual activity among high school girls in the study 
sample (59 percent) is higher than the percentage of Florida high school girls who had ever 
been sexually active in 2013 (40 percent).”” However, the rates of drug, alcohol, and tobacco 
use among the study sample are comparable to, if not lower than, those for the broader Florida 
female youth population. On average, among Florida high school girls in 2013, 34 percent were 
currently drinking alcohol and 20 percent were using marijuana, compared with 19 percent of 
high school girls in the study sample who said they were using drugs or alcohol. And 9 percent 
of Florida high school girls were smoking cigarettes compared with 12 percent of the study 
sample.” 


Many of the risk factors shown in these figures and the appendix tables overlap with 
those in the adverse childhood experiences (ACEs) literature. For example, ACEs include abuse 
or neglect, parental separation or divorce, and an incarcerated household member. ACEs re- 
search emphasizes prevention of these experiences as much as possible and links these experi- 
ences to future risky behavior, health problems, and early death, as well as other negative 
outcomes.” 


While many sample members had risk factors stemming from events or circumstances 
that occurred long before study enrollment or that had continued for some time, a significant 
portion of the study sample were experiencing recent changes or new situations in their lives 
shortly before their application to PACE and study enrollment. According to information col- 
lected by the PACE staff for a randomly selected sample of about 100 girls, about 35 percent of 
sample members applied to PACE because of recent experiences. Girls also talked about such 
experiences in interviews with the research team; some examples of tipping points that brought 
girls to PACE are detailed in Box 1.1. Many girls discussed a recent death of a family member or 
friend, getting poor grades or failing in school, recent expulsions or suspensions, or a recent al- 
tercation or fight. 


2K ann et al. (2014). 
3Kann et al. (2014). 
4Centers for Disease Control and Prevention (2016). 
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Box 1.1 
Examples of Why Girls Came to PACE 


Academic issues 


The girl was enrolled at a public high school but rarely attended. Her grades were very 
bad. After she got into a fight, the school encouraged her to “consider other options.” 


She was enrolled at a public high school but behind in her credits and wanted to catch up. 
Additionally, she was getting in trouble (not specified). 


She was in eighth grade for the third time and failing. She alluded to an “incident” that oc- 
curred the previous year but did not elaborate. 


She was very far behind in credits. The principal at her public high school told her that she 
would need three additional years to finish unless she went to another type of school. 


Behavioral issues and juvenile justice involvement 


The girl is a current DJJ client with six misdemeanors. She was recently suspended from 
school and had been suspended the previous semester for fighting. She often doesn’t go to 
school if she doesn’t feel like it. 


She fell in with the “wrong crowd” at her high school and started smoking marijuana; as 
she smoked more, her grades dropped further. She was also struggling with grief after the 
deaths of two of her friends in the past year. 


She was getting in trouble at school because she frequently skipped class. She was also 
disrespectful to teachers. 


She had been expelled from her previous school and hadn’t been attending for three 
months before that. She committed a misdemeanor and had to go to court. While at court, 
she learned about PACE from a friend who was also there. 


She was in tenth grade at a public high school for about three weeks before she got into a 
fight and was expelled. 


Bullying 


She was attending a new school after leaving her public middle school for various reasons. 
However, she did not like the new school because one of her classmates there had previ- 
ously bullied her at the public school. 


Though she was doing well in school academically, she was getting bullied. She learned 
about PACE from her older relative, who had attended and was now in college. The prin- 
cipal and assistant principal wanted her to stay in school; her mom had to get involved to 
withdraw her so she could enroll at PACE. 


(continued) 
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Box 1.1 (continued) 


Other 


¢ She was out of school for the first few weeks of ninth grade because of her sibling’s death. 
It was a difficult time for her family. She had not enjoyed her previous year at school; it 
had been a “bad year.” Her mom found PACE online. 


She had several incidents over two years that overwhelmed her. These included unex- 
pected deaths of close relatives, including a parent, and a move to Florida. 


SOURCES: PACE management information system and MDRC site visit interviews with study 
participants. 


Overview of the Report 


As the remainder of the report lays out, the evaluation found that PACE is a well-implemented, 
theory-based program model that positively affects girls at risk of juvenile justice involvement in 
the short term, leading them to be more engaged and on track academically. Chapter 2 describes 
the contrast between the services received by the program and control groups. Chapter 3 presents 
impacts on academic, youth development, risky behavior, and delinquency outcomes for the 12- 
to 18-month period following random assignment; information from semistructured interviews 
supplies context for these findings. Chapter 4 presents the cost analysis. The final chapter provides 
information on PACE today, summarizes the key findings, and discusses the implications. 
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Chapter 2 


PACE and Alternative Services 


This chapter describes the services the program group and control group reported receiving 
during the year following random assignment and compares differences in service receipt be- 
tween the two groups. Understanding the extent to which each group received services, and the 
characteristics of those services, can help with the interpretation of observed impacts, or the 
lack thereof, that will be presented in the next chapter. First, the chapter briefly describes PACE 
services. The next section presents impacts on service receipt across the two research groups, 
followed by an in-depth discussion about the characteristics of control group services that draws 
on data from the follow-up survey of girls and in-depth interviews with girls and parents. (See 
Appendix B for an analysis of survey response bias and Appendix D for information about the 
interview data collection.) 


The findings in this chapter indicate that, in general, the program group received more 
services and received them more often from a professional source, such as a counselor or aca- 
demic adviser, rather than a friend or family member. Nonetheless, the control group was likely 
to be enrolled in school, as expected given compulsory education requirements, and moderately 
engaged in receiving other academic and social services. Overall, the girls in the control group 
were unlikely to receive the full suite of PACE’s core services, and they were unlikely to receive 
services that had certain characteristics of high-quality programs for young people — an emphasis 
on family engagement, the presence of supportive adults, and follow-up services. However, 
among the minority of control group girls who did receive services similar to those provided by 
PACE, many of those services did have those characteristics. 


PACE Services 


Girls enrolled at PACE attend daily, during normal school hours. They receive a combination of 
academic and social services during the day. PACE, a year-round program, is designed holisti- 
cally, intending to provide “wrap-around” support for academic and social needs in one location. 
The program’s integration of academic and social services, embedded in a gender-responsive 
program culture, distinguish it among other programs for at-risk girls in Florida. An earlier report, 
Helping Girls Get Back on Track, goes into depth about the PACE model and the services it 
provides. ! 


The primary goal of academic services at PACE is to provide girls with the opportunity 
to earn credits and progress toward completion of middle school and high school. The academic 
curriculum at PACE follows state standards and mirrors that of public schools in Florida. When 
a girl enters PACE, the staff reviews her school transcripts and test scores to determine her aca- 
demic level and credit needs. Since girls enter the program at different points in their education 


'Treskon, Millenky, and Freedman (2017). 
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and need to complete different courses, girls in the same classroom are often working on different 
subjects. For example, half of a social studies class may be working on world history while the 
other half is focusing on U.S. history. PACE classes, capped at 12 students, are typically smaller 
than classes in the public schools.” The implementation study found that the quality of classroom 
instruction at PACE was similar to that of public school classrooms.? 


Girls at PACE receive academic advising every other week from an assigned teacher or 
guidance counselor. Advising meetings cover grades, progress toward earning credits, and test 
results. 


To promote their well-being and transition to adulthood, PACE offers girls life skills ed- 
ucation, including health topics such as physical and reproductive health, substance abuse pre- 
vention, avoiding delinquent behavior, and managing stress. At most PACE centers, girls attend 
life skills class, which is called Spirited Girls!, on a regular basis. PACE also provides opportu- 
nities for career exploration and volunteer service. Girls take a career assessment and learn about 
possible career options, and the program works to strengthen girls’ academic and “soft” skills 
(interpersonal skills and good work habits) to support work readiness. Girls also participate in a 
volunteer service project each semester to promote self-esteem, build work readiness skills, and 
contribute to the community. 


Through social services, PACE strives to address the nonacademic issues that may hinder 
a girl’s success in the program and beyond. Upon enrollment, social service staff members con- 
duct a comprehensive assessment and develop an individual care plan for each girl. Throughout 
her stay at PACE, social service staff members meet with each girl at least every other week to 
assess her progress and connect her or her family with appropriate support. All PACE centers 
have partnerships with outside providers to connect girls and families to other services, such as 
health services. For many girls, reviewing progress with their counselors was a jumping-off point 
for discussions about other issues, such as behavior, family concerns, or academic engagement. 
Many sessions covered multiple topics, and counselors often worked on strategies for girls to use 
to address the challenges they faced. 


Parental engagement is also a key feature of social services at PACE. Staff members 
conduct a home visit upon a girl’s enrollment in PACE and have monthly meetings with parents 
to update them on her progress. Once a girl is ready to leave PACE, social service staff members 
engage her and her family in planning for the transition, including helping to identify an appro- 
priate placement. After the girl leaves PACE, she receives 12 months of follow-up services, typ- 
ically phone calls from a designated staff person to check on how she is doing and, if needed, to 
connect her with additional services. 


The Florida average is 16 students per teacher, though this estimate includes elementary schools, which 
typically have smaller class sizes than middle and high schools. The average is calculated by dividing student 
membership by teachers for traditional schools (Florida Department of Education 2015a). 

3PACE classrooms were evaluated using the CLASS-S scoring tool. Additional information about the find- 
ings is available in the implementation report (Treskon, Millenky, and Freedman 2017). 
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PACE’s gender-responsive culture underpins the academic and social services it pro- 
vides. As discussed in Chapter 1, such a culture incorporates a focus on safety and high-quality 
relationships, a strengths-based approach, and an understanding of trauma in all aspects of pro- 
gram delivery. At PACE, staff relationships with girls are seen as central to implementing a gen- 
der-responsive approach and also to maintaining safety. Staff training and low staff-to-participant 
ratios help promote nurturing relationships between staff members and girls, and social service 
staff members work with girls on strategies to promote healthy relationships with their families 
and their romantic partners. Staff members also focus on developing girls’ self-confidence by 
identifying and building on their individual strengths. This focus on strengths is formalized 
through a rewards system, in which girls are granted tangible rewards or privileges for completing 
specified milestones. And finally, PACE staff members are trained to understand the impact of 
trauma on a person’s life, recognize the symptoms of trauma, and interact with girls in a way that 
avoids further traumatizing them and supports their healing. Illustrating the model in practice, 
Box 2.1 shares an example of one girl’s experience at PACE. 


Impacts on Service Receipt 


As a result of random assignment, girls assigned to the control group were not able to enroll in 
PACE and had to find other services in the community.* Many PACE centers referred control 
group girls to other services based on what they determined would be a good fit for the girl based 
on her intake assessment. 


Figure 2.1 presents impacts on service receipt for the 12 months following study enroll- 
ment, using the survey data. For some services, girls who reported receiving a service were asked 
from whom they received the service, and throughout this section we distinguish between services 
that are provided by a professional source and those that are not — for example, those received 
from a parent, other family member, or friend. This distinction was made to assess the extent to 
which the control group received services like those offered by the staff at PACE. A service pro- 
vided by a staff member at an organization or school is probably closer in content to the services 
received by girls at PACE than a service provided by a parent, family member, or friend. Appen- 
dix Table E.1 provides more details on the frequency of services received and the primary source 
reported by girls for many of these services. For some social services, only the control group was 
asked about the source; those measures are presented in Appendix Table E.2. 


Academic Services 


As shown in Figure 2.1, most members of the research sample reported that they attended 
a school or other academic program at some point during the follow-up period; the program group 


4As discussed in Appendix A, random assignment took place only when a center generally had more eligible 
applicants than slots available. 


19 


Box 2.1 


One Girl’s Experience at PACE 


Although there is no typical PACE girl, the following vignette is one example of a girl’s ex- 
perience at a PACE center. J, a 16-year-old expelled from her public high school for fighting 
with other students, learned about PACE from the school’s vice principal. It was the only 
program she considered once she was expelled. Every day, her brother drives her to PACE 
and she takes two buses home in the afternoon; the return commute takes about an hour 
and a half. 


J has found that the teachers provide a lot of assistance, and the classes are smaller than those at 
her prior school. She explains, 


These teachers, they help you out more than regular teachers at other schools 
... the teachers are nicer. They ask you if you need help, instead of you asking 
them. It’s just better than regular schools. . . . [The teachers] give me the work, 
and it’s hard, but they literally will help me find the answers . . . they will lead 
me to the answers and help me out. 


On the social services side, J estimates that she meets with her counselor five times every week. 


Whenever I have problems with other students, that’s when I go straight to 
[my counselor] and I talk to her about it... . She helps me with all my prob- 
lems. If I need help with work she’ll find me a tutor, if I’m having trouble at 
home she’! talk to me about it. 


One day, J had a conflict with another girl at the center, leading to a mediation session. Both 
girls and two counselors participated in the mediation, and each girl had a chance to tell her side 
of the story. This approach allowed the girls to resolve their conflict. 


Although initially J was worried about going to an all-girls program, she has found the other 
girls supportive. The absence of boys has prevented conflict and allows for more focus during 
classes. When she overdosed due to her struggle with depression, her friends at PACE let her 
know how much she was missed while absent from the center. 


I didn’t come for three weeks because of something that was going on, because 
I OD’d. I was really depressed, and that’s when my friend was like “we’re 
here to talk. We’re mad at you because you didn’t talk to us, we’re here for 
you, we want you to be successful in life and come to school every day. We 
don’t want you to miss out on anything.” 


J says things are better for her since she came to PACE. She doesn’t get into fights as much, and 
she no longer feels like the “bad child.” She is closer to her family and makes an effort to have 
real conversations with her family members. J plans to go to a different public high school when 
she leaves PACE and aspires to attend college. 
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Figure 2.1 


One-Year Differences in Service Receipt 


Percentage Percentage point 


point difference in 
i difference in receipt from 
Key services service professional 
Percentage who received service receipt source 
Ever enrolled in schoo! | 
or education program in Bee = 
past year 
Nonprofessional source Professional source 
Vv Vv 
Received academic ’ He 418" 33*** 
advising 30 Wh 
Received help planning 13 Ra ee 
’ 5 29 
for a job or career 
Received counseling 19*** 20*** 
or therapy 
Received help finding 
services in the 16*** 47*** 


community 


Other services? 


Received help planning 44*** 
one 


Received help related 


fo sonal, sono a 
ie 


reproductive health 


Received help related oe 
; : 13 
to social and emotional 
F 64 
skills 


ie Program group a Control group 


SOURCE: MDRC calculations based on responses to the PACE evaluation 12-month follow-up survey. 


NOTES: Results in this figure are regression-adjusted, controlling for pre-random assignment characteristics. 
Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 
Rounding may cause slight discrepancies in sums and differences. 
alt was not possible to tell whether the respondent received these services from a professional or 
nonprofessional source. 
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was slightly more likely (98 percent versus 93 percent) than the control group to have been en- 
rolled in an educational program.° The high enrollment rates are not surprising given the age of 
participants — in Florida, children under 16 must be enrolled in school.® Most program group 
members reported that they received academic services from PACE. Control group members re- 
ceived services primarily from a local public or charter school. A small number of control group 
girls (estimated at about 15 percent) attended an alternative school. Such schools, operated either 
by the district or a contracted provider, serve students who have not been successful in the tradi- 
tional school setting; some offer additional supportive services to help students be successful. 


Academic advising includes activities that relate to short-term academic goals, like 
building a class schedule that includes credits needed for grade promotion. At PACE, as noted 
above, an assigned teacher or guidance counselor provides academic advising twice a month. 
Program group girls were more likely to report having recetved academic advising than girls in 
the control group. As shown in the top panel of Figure 2.1, the program group was also more 
likely to have received advising from a professional source (usually PACE), and they received 
the service more frequently. Similarly, as shown in Appendix Table E.1, program group girls 
were more likely to have received career planning assistance from a professional source (again, 
mostly from PACE). About half the control group respondents who received academic advising 
reported that they received it from someone at school, but a parent, guardian, or other relative 
was the source of support more than 40 percent of the time (among those who received aca- 
demic advising). In career planning, the control group reported that a parent or guardian was 
the most common source of support. 


Social Services 


The girls’ risk factors at the time of study enrollment indicated that many of them could 
benefit from social services to address their behavioral health and basic needs. As shown in the 
bottom panel of Figure 2.1, the program group was more likely to report that they received social 
services than the control group. The program group reported that they received these services 
most often from PACE, which is not surprising given that counseling, case management, repro- 
ductive health education, and social-emotional skill development are provided to all girls who 
attend PACE. 


The largest difference in service receipt between the two groups was for counseling or 
therapy. The program group was much more likely to receive this service, and they received the 
service more frequently. Though the majority of program group girls reported receiving this ser- 
vice from PACE, more than one-third reported receiving services elsewhere, most commonly at 
a center, clinic, or private practice (Appendix Table E.1). Girls sometimes came to PACE already 
having an outside counselor or therapist, and in other cases, PACE determined that the staff could 
not meet the girls’ needs and provided referrals, so this finding is not unexpected. Most of the 


>The administrative data from the Florida Department of Education told a similar story, though participation 
rates were slightly lower for both groups: 94 percent of the program group and 90 percent of the control group 
were ever enrolled in school. 


Florida Department of Education (2018a). 
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control group members who received counseling or therapy reported receiving it from a health 
care center or independent clinician, while a much smaller share reported receiving the service at 
school. Like the program group, control group girls may have already been connected to these 
services before they applied to PACE. 


There was a similar difference in whether and how girls received help finding other ser- 
vices in the community, such as services to meet basic needs like housing, health care, or trans- 
portation. About one-third of the program group received such referrals, while about one-fifth of 
the control group did. As shown in Appendix Table E.1, the majority of program group girls 
received help with referrals from PACE; the control group received it most commonly from 
someone at school or through someone connected to the Department of Juvenile Justice. 


The survey also measured the difference in service receipt related to the type of repro- 
ductive health and social-emotional competence education that is part of the PACE Spirited Girls! 
curriculum. Program group girls reported high rates of receiving these services, but control group 
rates were also high (Figure 2.1). The survey did not collect information on the source of these 
services for the program group, but given that these services are a core component of PACE’s 
model and the program group attended PACE at high rates, it is assumed that the program group 
for the large part received these services from PACE. The control group, who did not have access 
to services provided by PACE, reported most often that they received help with reproductive 
health or social-emotional skills from a parent, guardian, or other relative. About one-quarter of 
control group girls surveyed reported that they received such services at school (see Appendix 
Table E.2). 


Combination of Services 


For PACE, and other programs that provide comprehensive services in one location de- 
signed to meet the holistic needs of participants, providing those services together is a core aspect 
of the model. This approach promotes communication between service providers; for example, 
teachers and counselors can share information about emerging issues in the classroom or in a 
girl’s life. Integrating services may improve participation by reducing the burden on girls and 
families to travel to multiple service providers. 


Data from the survey reveal the extent to which the program group and the control group 
received a package of core services like that provided by PACE. To analyze the combination of 
services the girls received, five “key services” that PACE provides were selected from the survey 
measures: academic classes, academic advising, job or career planning, counseling or therapy, 
and case management. This analysis counts a key service only if the sample member reported 
receiving it from a professional source. As Figure 2.2 shows, the program group reported receiv- 
ing all five key services more frequently than the control group. Twenty percent of the program 
group received all five services, compared with just 3 percent of the control group. Two-thirds of 
the control group reported that they received two or fewer services, compared with one-third of 
the program group. 
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Figure 2.2 


Receipt of Key Services, by Research Group 
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SOURCE: MDRC calculations based on data from the PACE evaluation 12-month follow-up survey. 


Still, even though PACE provides all services in one place, about half the program group 
members reported that they received three or fewer services. This may have to do with the girls’ 
perceptions; for example, they may not consider their PACE counselor to be a counselor or ther- 
apist.’ Also, some program group girls attended PACE for a short time and did not receive all its 


TAs described in the implementation report, PACE counselors varied in their approach to working with the 
girls, with some taking on a more therapeutic role and others focused more on care planning. Counselors would 
also vary their approach based on the needs of the girls, so girls who did not have active mental health needs 
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services. PACE’s management information system data indicates that those who participated in 
PACE for at least 30 days did receive most of the components of the model.® 


Qualities of Alternative Services 


The findings presented above focus on the rates at which girls participated in services and the 
source of some of those services, but they provide limited information about the content or quality 
of those services. The in-depth implementation study included in this evaluation provided infor- 
mation about the content of PACE services and some aspects of program quality. Since control 
group members attended a vast array of different schools and services, their content and quality 
could not be assessed individually. Yet content and quality are key to interpreting impacts. To 
provide insight into some aspects of those services, this section presents findings from an extra 
survey module administered to the control group that asked about services provided by a profes- 
sional source. This section also draws on phone interviews that MDRC conducted with a subset 
of control group girls and parents a little more than a year after study enrollment.’ 


Academic Services 


As Figure 2.3 shows, the survey asked girls in the control group about aspects of the 
schools or educational programs they attended that might be comparable to key features of 
PACE’s program. Unsurprisingly, given the small number of all-girls education providers in Flor- 
ida, few members of the control group attended an all-girls school. But close to half the girls 
reported that their school offered incentives or rewards for reaching academic or behavior goals. 
More than one-third of girls surveyed said that someone at their school had connected them to 
support programs outside of school when needed, and nearly three-quarters of girls surveyed said 
that there was someone at the school to whom they could turn for advice about personal problems. 


Girls who participated in the phone interviews described a variety of educational path- 
ways after they were assigned to the control group, with many noting that they had attended a 
few different schools or programs since random assignment. Most girls were at a public middle 
school or high school at some point, but some control group girls reported that they attended an 
alternative or specialized school. Girls interviewed expressed a wide variety of experiences in 


probably would not have covered counseling or therapy topics in their meetings with PACE counselors (Treskon, 
Millenky, and Freedman 2017). 

8As reported in Treskon, Millenky, and Freedman (2017), 93.5 percent of PACE girls received academic 
advising, and 97 percent received counseling, which would have also included case management. All girls en- 
rolled in PACE attend Spirited Girls!, which includes job and career planning, topics which may also be covered 
in counseling. Girls who stayed at PACE for more than 30 days participated in an average of about 3.7 key 
services, compared with 3.2 services for those who had a shorter stay. 


°See Appendix D for details about how the girl and parent interviews were conducted. For the control group, 
25 girls and 18 parents were interviewed. Program group girls were also interviewed and asked similar questions; 
their responses were included in the earlier implementation report (Treskon, Millenky, and Freedman 2017). 
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Figure 2.3 


Aspects of Academic Services Received by the Control Group 
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SOURCE: MDRC calculations based on responses to the PACE evaluation 12-month follow-up survey. 


44% 


these schools, ranging from very positive reflections on the school’s support for them and their 
needs to very negative descriptions of their experience with school staff members or other stu- 
dents. Girls interviewed were also split on whether they felt safe at their schools, with some feel- 
ing safe and some describing fights or other unsafe situations. !° Many of the control group parents 
interviewed had concerns about safety at their daughters’ schools. This is in contrast to program 
group parents, who, as reported in the implementation report, emphasized the safety of PACE as 
a draw of the program. '! 


Control group girls who were interviewed often described participating in credit-recovery 
or virtual-school programs at some point during the follow-up period. These included stand-alone 
programs offered by charter or alternative schools, as well as programs within traditional schools 
designed for students who were behind on credits. Girls interviewed reported mixed experiences 
with online learning. With self-paced learning, some girls found it challenging to stay motivated 
and get support when they needed it. Others found that the approach worked for them to catch up 
on credits. 


Control group parents who were interviewed generally reported regular communications 
with the schools about their daughters’ progress. Communication tended to be either at structured 
times (monthly meetings, for example) or when there was a problem with grades or behavior, and 
these communications were often initiated by the school. Some parents reported that the school 
offered an online portal where they could monitor the child’s progress. 


The control group module asked girls whether they received information about other 
topics that would be covered in PACE’s Spirited Girls! curriculum (in addition to the social- 


‘0A bout a quarter of control group girls surveyed reported that they felt unsafe at times at school. 
''Treskon, Millenky, and Freedman (2017). 
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emotional and reproductive health services discussed above): healthy lifestyles, anger manage- 
ment, substance abuse, avoiding delinquent behavior, and women’s issues. Figure 2.4 shows 
that one-fifth to half of the control group girls received education in these areas. In the inter- 
views, girls said that these topics were sometimes covered in health classes at school, but they 
did not seem to be a key feature of the schools the girls attended. 


Figure 2.4 


Topics Covered in Services Received by the Control Group 
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SOURCE: MDRC calculations based on responses to the PACE evaluation 12-month follow-up survey. 


Engagement of Parents 


To be able to draw comparisons to the parental engagement component of PACE, control 
group girls surveyed were asked whether parents or guardians attended meetings with them at the 
places where they received services. Although this measure does not offer a direct comparison to 
the monthly parental engagement that PACE strives for, it does provide information about 
whether the service provider tried to engage parents. The left side of Figure 2.5 shows that for 
most services, the majority of girls who received a service reported that a parent or guardian 
attended a meeting about it. Parents were most likely to be engaged by a counselor or therapy 
provider and least likely to be engaged for job and career planning. The differing levels of en- 
gagement by service type make sense in the context of the specific services — counseling and 
case management services often have a parental consent component that serves as a lever for 
parental engagement. However, when the measures are viewed over the full sample, as shown on 
the right side, only a small portion of the sample received professional services with parental or 
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Figure 2.5 


Parental Engagement in Control Group Services 
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NOTE: Measures refer to services provided by a professional source, which includes someone other 
than a parent, guardian, other relative, or friend. 


guardian involvement. In contrast, it is expected that most girls who attended PACE experienced 
such involvement. 


Presence of a Supportive Adult 


Building healthy and trusting relationships between girls and PACE staff members is a 
part of the PACE model, and research on youth programs has emphasized the important role of 
supportive adults in a young person’s life.!? Nearly three-quarters of control group girls reported 
in the follow-up survey that they had someone at their schools they could turn to for advice about 
personal problems (see Figure 2.3). Additionally, the majority of girls who reported receiving the 
academic and social services shown in Figure 2.6 reported having an adult who cared about them 
at the place where they received services. This probably reflects the expected qualities of organi- 
zations that serve young people. Still, not all girls felt that they had an adult to talk with. One 
control group girl interviewed described her feeling about the counselors this way: “She was as- 
signed to me because of my last name. ... They’re there to get paid and I’m not going to waste 
my time on people who don’t care.” But other girls described having a caring teacher or counselor 


Treskon, Millenky, and Freedman (2017). On average, PACE staff members attempted to contact parents 
or guardians two and a half times per month. Not all contact attempts were successful. 


Murphey, Bandy, Schmitz, and Moore (2013). 
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Figure 2.6 


Presence of a Supportive Adult in Control Group Services 
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SOURCE: MDRC calculations based on responses to the PACE evaluation 12-month follow-up 
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than a parent, guardian, other relative, or friend. 


that they could rely on for support, and some girls reportedly were still in contact with that person. 
As with parent or guardian engagement, the right side of Figure 2.6 shows that a minority of the 
full control group sample received services with a supportive adult available. 


Follow-Up Services 


Girls who have been at PACE for longer than 30 days are eligible for follow-up services 
for up to a year after they leave the program. Follow-up services are a way to support girls as they 
make the transition to a new program by providing continuity of care and helping to identify 
emergent needs or additional services that the girl might need. '* Many successful youth programs 
provide some type of follow-up services.'* Control group girls were asked whether the places 
they received services called to check up on them after they stopped receiving services. As shown 
in Figure 2.7, follow-up services were reported by about 40 percent of girls who had received 
counseling and job and career planning services, and by more than half the girls who had received 
academic advising and case management (connection to services in the community). As with the 


'4Follow-up services were an area where implementation was more uneven across the PACE centers. Al- 
though a few centers described robust services, most did the minimum — calling girls at the required intervals 
and connecting girls who sought help with supportive services (Treskon, Millenky, and Freedman 2017). 


'SMiller et al. (2016); Fein and Hamadyk (2018); Millenky, Bloom, Muller-Ravett, and Broadus (2011). 
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Figure 2.7 


Receipt of Follow-Up Services in Control Group Services 
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NOTES: Measures refer to services provided by a professional source, which includes someone other 
than a parent, guardian, other relative, or friend. Measures do not include girls who were still receiving 
the service at the time of the survey. 


other measures, the portion of girls across the control group who received both the service and 
follow-up services was small. 


Conclusion 


The girls in PACE received more services than the girls in the control group, but the control group 
did attend school, and they took part in other academic and social services to a lesser extent. 
Notably, however, the control group received many of these services from nonprofessional 
sources, such as family members. They were less likely to receive the full suite of services that 
PACE provides, and few of the girls received professional services that included parental engage- 
ment, the presence of a supportive adult, or follow-up attention. 


The next chapter provides findings on academic and juvenile justice impacts in the 12- 
to 18-month period following random assignment; the findings on service contrast presented in 
this chapter provide a context with which to interpret those findings. 


Chapter 3 


Impacts on Girls’ Outcomes 


The previous chapter presented findings that girls in both groups received academic and other 
services in the 12-month period following their application to PACE. In the absence of PACE, 
the control group received many of these services from nonprofessional sources, such as a family 
member. Control group members were also less likely to receive the full suite of services that 
PACE provides and, for the most part, did not receive professional services that included parental 
engagement, a supportive adult, or follow-up services. 


This service contrast is important for the impact findings presented in this chapter. The 
evaluation’s key domains are academics, interpersonal relationships, risky behavior and juvenile 
justice involvement, and attitudes toward the future. (While the findings are presented by do- 
mains, the research design tests the effects of the holistic PACE model rather than individual 
components of the model.) Given the random assignment design of the study, statistically signif- 
icant differences — estimated effects that would be very unlikely to occur if the program truly 
had no effect — that emerge between the two research groups on these outcomes can be attributed 
to the PACE program. The differences discussed in this chapter are statistically significant. The 
analysis uses an “intent-to-treat” design, which includes all girls assigned to each research group 
in the analysis, regardless of whether they recetved PACE or other services, and compares out- 
comes between the two groups. 


The findings in this chapter provide information on the 12- to 18-month period following 
a girl’s application to PACE and enrollment in the study. For the girls in the program group, this 
follow-up period includes the months a girl attended a PACE center (an average of eight for those 
in the study) and could also include the months spent at another school or educational program. 
On average, the girls are between the ages of 15 and 16 during this period, but they range in age 
from 12 to 18. The impacts presented in this chapter draw on administrative records data and 
responses from the 12-month follow-up survey. About one-third of the girls in the program group 
indicated at the time of the survey that they were still enrolled at a PACE center. The chapter also 
presents select sample member and parent perspectives related to some key outcomes, based on 
telephone interviews with girls and their parents.! 


In sum, PACE led to some positive impacts on academic engagement, which includes 
measures of enrollment and attendance. No clear pattern of impacts emerged for youth develop- 
ment, risky behavior, or juvenile justice outcomes. As noted in Chapter 1, many youth develop- 
ment constructs are difficult to measure, and any impacts on formal juvenile justice involvement 
may take longer to occur. 


‘Semistructured interviews were conducted with a smaller, nonrandom sample of girls and their parents 
about one year after random assignment. See Appendix D for more about the interview structure and sample. 
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Overall, survey findings illustrate that sample members across both groups remain hope- 
ful about their futures one year after study enrollment. These findings suggest that girls’ lives 
have stabilized somewhat since the crisis state or tipping point that many sample members were 
experiencing at the time they applied to PACE. The control group outcomes can be seen as what 
would have happened to girls in the absence of PACE: Most enrolled in school during the aca- 
demic year but not over the summer, some were on track academically, few engaged in risky 
behavior, and the majority were focused on their futures. 


Educational Outcomes 


This section presents one-year impacts on educational outcomes, including the following 
highlights: 


e PACE increased school attendance. Girls in the program group were more 
likely to be enrolled for more days and have fewer absences than those in the 
control group. 


e Because of PACE, program group girls in high school were more likely to be 
“on track” academically than peers in the control group. 


e There were no statistically significant differences between the rates at which 
the program and control groups were promoted to the next grade level or the 
average number of credits earned. 


Academic instruction in a small class setting is an important component of the PACE 
program model. Many girls struggled academically before enrolling in PACE and came to PACE 
with the goal of catching up on credits. As shown in the logic model in Chapter 1, PACE’s 
holistic program model is structured to keep girls engaged — attending class and completing 
course work — and to further school progress. Research shows that outcomes such as attendance 
and course completion are key indicators of high school graduation. There is some evidence that 
behavior, as measured by disciplinary actions and behavior marks, is also predictive of high 
school graduation.* 


One-year impacts on educational outcomes from the Florida state-level administrative 
records are presented in Table 3.1. All outcomes reflect only activity that occurred within the 
Florida public school system. For program group girls, outcomes could reflect activity that oc- 
curred both at PACE and at another school or educational program. There are some additional 
limitations on how these outcomes can be presented, discussed further in Appendix F. 


?Treskon, Millenky, and Freedman (2017). 
Allensworth, Nagaoka, and Johnson (2018). 
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Table 3.1 
One-Year Impacts on Education Outcomes, from Administrative Data 


Program Control Difference 
Outcome Group Group (Impact) P-Value 


Academic engagement 


Ever enrolled (%) 93.9 89.6 44*** 0.010 
Number of days enrolled 136.7 123.3 13.4 *** 0.002 
Number of days present 119.6 109.8 9.8 ** 0.013 
Number of days absent 24.3 27.9 -3.6 ** 0.031 
Percentage of days absent 18.8 22.7 3:9" 0.004 
Ever enrolled in postsecondary classes (%) 2.4 1.4 1.1 0.236 
Ever enrolled in summer term (%) 39.4 12.6 26.8 *** 0.000 


Academic progress 
On track, based on composite measure (high school students)* (%) 27.6 14.2 13.4 *** 0.000 


Promoted to a higher grade” (%) 68.9 65.5 3.4 0.334 


Academic year (high school students) 
Total credits earned 3.8 3.7 0.1 0.696 
Credits earned (as % of credits attempted) 77.5 66.9 10.6 *** 0.001 


Summer (high school students) 
Total credits earned 0.3 0.0 0.2 *** 0.000 
Credits earned (as % of credits attempted) 18.9 2.9 16.1 *** 0.000 


Disciplinary actions (%) 


Ever expelled 0.5 0.6 -0.1 0.898 
Ever suspended 29.6 46.4 -16.7 *** 0.000 
Sample size 630 410 


SOURCE: MDRC calculations based on data from the Florida Department of Education. 


NOTES: All measures refer to outcomes only in the Florida public school system. 

Unless otherwise noted, measures refer to the full calendar year, which includes both the academic year and the 
summer term. The academic year refers to the traditional school term in Florida, which runs from about mid-August 
through the end of May. 

Results in this table are regression-adjusted, controlling for pre-random assignment characteristics. Statistical 
significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Rounding may cause slight discrepancies in sums and differences. 

@The composite includes the following criteria: absent less than 10 percent of days; never failed a core course; 
never expelled or suspended. Core classes include English language arts, math, science, or social studies. 

>For those in the last year of high school, graduation substitutes for promotion. 


33 


As shown in Chapter 2, girls in the program group were more likely to be enrolled in a 
school or educational program than control group girls. These findings are consistent with admin- 
istrative records: 94 percent of the program group and 90 percent of the control group were en- 
rolled within the Florida public school system in the first follow-up year. Florida law requires 
students to be enrolled until they reach the age of 16.4 Among the sample members who were not 
enrolled in the first follow-up year, about one-third were under the age of 16. 


In contrast, at the time of study enrollment, a considerably larger portion — 27 percent 
— of the study sample was not currently enrolled in school. This is a clear example of how girls 
applied to PACE at a time of crisis and then stabilized, even in the absence of PACE. 


PACE increased academic engagement by a statistically significant amount, as shown in 
Table 3.1. The program group was, on average, enrolled for about 13 more days than the control 
group and had fewer absences (19 percent of enrolled school days) than the control group (23 
percent). This translates to being present in school for two full additional weeks. There was also 
a substantial impact on enrollment in the summer (27 percentage points), which reflects PACE’s 
year-round program structure. This difference in summer enrollment is probably driving the im- 
pact on number of days enrolled, a calendar year measure.° Some research suggests that being 
out of school during the summer results in a learning loss for students, and that this loss is larger 
for lower-income students compared with higher-income students who may have access to addi- 
tional learning resources over the summer.° 


The administrative data also provide information on academic progress, including credits 
earned, grade promotions, and being on track to graduate.’ During the academic year, high school 
girls in the program group succeeded in earning a higher percentage of credits attempted than 
their peers in the control group — about 78 percent, compared with 67 percent for the control 
group, an impact of 11 percentage points — but did not appear to earn more credits overall.* 
PACE girls usually take fewer classes than girls at a traditional school.’ 


High school girls in the program group were also more likely to be on track academically 
in this follow-up year, as measured by a composite that defines a student as “on track” if she 
meets the following criteria: a high attendance rate, no expulsions or suspensions, and no core 


4Students age 16 and older who wish to stop attending school must file a formal declaration of intent to 
terminate school enrollment, which needs to be signed by a parent (Florida Department of Education 201 8a). 

>The full calendar year includes both the academic year and the summer term. The academic year refers to 
the traditional school term in Florida, which runs from about mid-August through the end of May. 

Quinn and Polikoff (2017). 

"Data on credits and course grades were available only for high school students, which is about half of the 
study sample. 

8As a point of reference, students in Florida need 24 credits to receive a standard diploma. A student com- 
pleting high school in four years, on average, completes 6 credits per year. The credits data presented in Table 
3.1 show the average credits earned for each group (3.7-3.8 credits). This includes 0 credits in the calculation for 
girls not enrolled, which brings down the overall average (Florida Department of Education 2016). 

°All core classes and a life skills class are offered to girls at PACE. PACE does not offer the electives found 
at traditional schools. 
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course failures.'° Across the full research sample, about half of the girls who were not on track 
did not meet at least two of these benchmarks. Most commonly, girls who were not on track had 
a low attendance rate. The on-track components reflect what researchers consider to be indicators 
of future high school graduation — attendance, behavior, and course performance — often re- 
ferred to in the field as the “ABCs.”"' There is no single set of thresholds across these indicators 
that is widely used in the field to measure the likelihood of high school graduation. 


There was no significant difference between the program and control groups in grade- 
level promotion. However, data on this outcome were more likely to be missing for the program 
group than the control group. This may be due somewhat to the relay of a girl’s school records 
from PACE to the school district. Specifically, PACE indicates to the school district whether a 
girl is eligible for promotion, but it is usually up to the school district — or an intermediary — to 
enter all the information about credits and course work and make an ultimate determination about 
promotion eligibility. This last step may not always occur in a timely fashion, resulting in the 
missing data. The 14 centers in the study are all located in different school districts with distinct 
processes for these data exchanges. Given these data limitations, the promotion measure pre- 
sented in this report is conservative and does not make any assumptions about promotions where 
the data are missing. 


The third panel of Table 3.1 presents school-based disciplinary actions. As noted at the 
beginning of the chapter, these disciplinary actions could have occurred either at PACE or at 
another school, during the follow-up period. Rates of expulsion were low across the study sample 
(less than 1 percent). For suspensions, the control group was more likely (with an impact of ap- 
proximately 17 percentage points) to have been suspended during the follow-up year than the 
program group. This could be due to PACE’s approach to working with girls; while PACE centers 
have policies governing behavior, girls are rarely suspended or expelled. Instead, a girl who does 
not comply with a center’s policies may be asked to take one or more “days of reflection” in or 
out of the PACE center. These days of reflection are usually not recorded on a girl’s record as a 
suspension. ! 


Table 3.2 presents one-year impacts on academic engagement based on responses to the 
12-month follow-up survey. Overall, the impacts based on the follow-up survey align with those 
from administrative records, though the follow-up periods for these outcomes differ slightly (see 
Appendix F for more information). Suspensions are one outcome where the information reported 
in the two data sources did not line up. In the survey, girls in the two research groups reported 
similarly on whether they had been suspended in the year since study enrollment (28 percent to 
30 percent). One possible explanation for the discrepancy between the control group self-reported 
measure and the one based on administrative records is that control group girls did not report 


'OCore courses include English language arts, math, science, or social studies. High attendance is considered 
being absent less than 10 percent of days enrolled. 
'l Allensworth, Nagaoka, and Johnson (2018). 


"Reporting requirements regarding suspensions and expulsions differ by school district. 
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Table 3.2 
One-Year Impacts on Academic Progress and Engagement, from Survey 


Program Control Difference 


Outcome Group Group (Impact) P-Value 
Months enrolled in a school or educational program 9.7 8.9 0.8 *** 0.003 
Currently enrolled in a school or educational program (%) 82.1 79.6 2.5 0.356 
Number of unexcused absences in past month 2.2 3.0 -0.8 ** 0.039 
Ever suspended from school (%) 30.0 27.8 2.2 0.474 
Number of suspensions 0.7 0.8 -0.1 0.441 
Sample size 529 341 


SOURCE: MDRC calculations based on responses to the PACE evaluation 12-month follow-up survey. 


NOTES: Results in this table are regression-adjusted, controlling for pre-random assignment characteristics. 
Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 


in-school suspensions in the survey, while the measure based on administrative records includes 


such suspensions. 


Interpersonal Relationships 


This section presents impacts related to interpersonal relationships based on data from the 12- 


month follow-up survey. Overall, girls in the study sample — across both groups — appeared 


to 


have positive relationships and found a level of support in the home and beyond. No clear pattern 
of differences emerged between the program and control groups. Box 3.1 provides some sample 


members’ perspectives on interpersonal relationships, based on information collected in a set 
separate, semistructured telephone interviews. 


of 


As discussed in Chapter 1, PACE focuses on improving girls’ interpersonal skills and 
encouraging healthy relationships. Attachment, or the persistent bond between parent and child, 
can serve as a protective factor for children and adolescents, contributing to more positive social- 
emotional outcomes.'? In addition, parental engagement is a central component of the PACE 


model (and gender-responsive programming, more broadly). 


Bowlby (1983); Moretti and Peled (2004). 
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Box 3.1 


Girls’ Perspectives on Interpersonal Relationships 


On family members 


Most girls interviewed indicated a close attachment to their mothers, though relationships had 
both positive and negative emotional undercurrents. One girl noted that “mothers and daughters 
always bump heads” while another shared the fact that she talks to her mother about “anything 
and everything you can think of.” 


On PACE staff members 


In interviews, girls who attended PACE overwhelmingly indicated that they felt comfortable 
talking to PACE staff members about their problems. One girl, who was pregnant during her 
time at PACE, said that a PACE staff person was “there throughout my whole pregnancy. .. . 
She was helpful because I didn’t have anybody. My family, they didn’t very much approve of 
that [the pregnancy].” 


On friends 


Girls in both groups tended to have a moderate number of friendships. In interviews, program 
group girls expressed having one or a few friends at PACE, though very few PACE girls de- 
scribed close relationships or a multitude of friendships at PACE or at other schools. A few girls 
noted that they preferred to avoid bad influences. One control group girl said, “I realized I have 
bigger things to worry about than petty people who are just going to show me the wrong things.” 
Fewer than a dozen girls in both groups indicated that they had no close friendships at all. 


SOURCE: MDRC telephone interviews with study participants. 


Table 3.3 indicates that, at the time of the 12-month survey, the vast majority of girls in 
both research groups reported living in their parents’ homes, as they did when they applied to 
PACE (see Figure 1.3). To assess more specific characteristics of girls’ relationships with their 
mothers, such as trust and feelings of alienation, the survey asked girls questions from the Inven- 
tory of Parent and Peer Attachment.'* Across both research groups, responses indicated that the 
girls had general feelings of attachment to their mothers and felt accepted by them.!° 


Table 3.3 presents similar rates of girls having a supportive adult (other than a parent or 
guardian) in their lives since study enrollment. Supportive nonparenting adults provide adoles- 
cents the opportunity to share information they may not be comfortable telling their parents and 
to seek additional resources or advice that might otherwise be unavailable. These adults may fall 


'4 Armsden and Greenberg (2009). 


‘Table 3.3 presents average scores and effect sizes based on girls’ responses to these questions. The effect 
size indicates PACE’s effect as a proportion of the standard deviation of the outcome. This metric allows for 
comparisons of impacts across studies and provides some indication of the magnitude of the impact. 
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Table 3.3 


One-Year Impacts on Youth Development, 
Social Support, and Interpersonal Relationships 


Program Control Difference 
Outcome Group Group (Impact) P-Value 


Social support and interpersonal relationships 


Currently living in parent's home (%) 83.8 84.4 -0.6 0.821 
Has supportive adult in her life* (%) 60.9 61.1 -0.2 0.945 
Mother relationship score? (average) 3.7 3.7 0.1 0.417 


Relationships with friends (%) 


Feels understood by friends 88.4 90.2 -1.9 0.394 
Friends help her understand herself better 82.4 83.6 -1.2 0.643 
Friends care about her feelings 92.4 94.8 -2.5 0.159 
Confides in her friends 86.0 89.9 -3.9 * 0.094 
Friends show her concern 92.1 93.0 -0.8 0.647 
Been in a violent or abusive romantic relationship (%) 6.4 5.0 1.4 0.383 


Youth development 
Social competence score* (average) 2.9 2.9 0.0 0.692 


Reaction to problems or arguments with others (%) 


Reflects and thinks about what went wrong 79.4 79.9 -0.5 0.853 
Talks to other person to straighten it out 66.6 69.0 -2.4 0.453 
Sample size 529 342 


SOURCE: MDRC calculations based on responses to the PACE Evaluation 12-Month Survey. 


NOTES: Results in this table are regression-adjusted, controlling for pre-random assignment characteristics. 
Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

aOther than parent or guardian. 

Score is calculated as an average of responses to a subset of items on the Mother subscale of the Inventory of 
Parent and Peer Attachment. The possible score range is 1 to 5. The standard error is 0.063 and the effect size is 
0.057. 

°Score is calculated as an average of responses to a subset of items on the Social Competence subscale of the 
Harter Self-Perception Profile for Adolescents. The possible score range is 1 to 4. The standard error is 0.053 and 
the effect size is 0.027. 


in a variety of categories, some professional and some not, and these relationships are associated 
with such favorable outcomes as higher academic achievement, more positive attitudes about 
school, higher self-esteem, and a decrease in behavior problems, such as substance abuse.'° 


‘6Sterrett, Jones, McKee, and Kincaid (2011). 
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Among the program group, 45 percent of girls identified a staff member from PACE as a sup- 
portive adult in their lives. A professional may be better equipped to provide sound advice and 
have knowledge of other resources. 


As shown in Table 3.3, at the time of the survey girls in both groups appeared to have 
solid friendships with one or more peers. Across the full sample, girls reported that their friends 
cared about them and they confided in their friends, though few statistically significant differences 
emerged between the program and control groups. Another set of questions in the survey used a 
scale to measure social competence at the time of the survey; that is, the ability to make friends 
and successfully interact with peers in social settings.'’ The average scores for the sample on the 
social competence scale are presented with the corresponding effect size. As shown, the program 
and control groups had similar scores on this measure of social competence. 


Risky Behavior and Juvenile Justice 


This section presents impacts on risky sexual and delinquent behaviors, including formal involve- 
ment with the juvenile justice system in Florida. Both groups exhibited relatively low levels of 
risky behavior, given the higher risk at the time of enrollment. There were few statistically sig- 
nificant differences between the program and control groups on these self-reported measures of 
risky behavior or on formal juvenile justice involvement based on data from the Florida Depart- 
ment of Juvenile Justice. 


Girls come to PACE with backgrounds and characteristics that put them at risk for future 
involvement in the juvenile or criminal justice systems, as discussed in Chapter 1. These charac- 
teristics include low school engagement, a history of delinquency, a history of abuse or neglect, 
and behavior such as drug, alcohol, and tobacco use. The PACE program model is designed to 
address these risk factors and reduce such behavior among girls. As shown in the logic model 
(Figure 1.1), on average, PACE could help lower delinquent behavior in the short term and lower 
involvement in the justice system in the longer term. 


Risky Behavior 


As shown in Table 3.4, PACE had no significant impacts on binge drinking. Sample 
members’ reports of this behavior (8 percent of the program group and 6 percent of control group) 
were lower than among a broader population of girls: Outside surveys report that 13 percent of 
girls in Florida and 17 percent of girls nationally engage in binge drinking. '* Girls in the program 
group were more likely (by 3 percentage points) than those in the control group to report any 
illegal drug use over the past year. !” 


Harter (2012). 


'8Kann et al. (2016). The study survey measure covers the past two weeks, and Florida and national citations 
refer to the past 30 days. 


‘Girls were specifically asked about the use of cocaine, inhalants, and pills without a doctor’ s prescription. 
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Table 3.4 
One-Year Impacts on Risky Behavior 


Program Control Difference 


Outcome (%) Group Group (Impact) P-Value 
Substance use and delinquency 

Binge drank in the past two weeks? 7.7 6.2 1.5 0.415 
Used illegal drugs” 10.4 7A 3.3 * 0.093 
Drinking or drug use interfered with responsibilities 9.1 6.4 2.8 0.141 


Involved in any delinquent incidents in past 3 months 
Any violent incidents 22.7 25.4 -2.7 0.345 
Any property incidents 40.0 49.8 -9.8 *** 0.004 


Sexual activity® 
Sexually active and did not use pregnancy protection 


method during last sexual encounter 8.8 8.6 0.3 0.889 
Sexually active and had more than one partner 8.7 6.1 2.6 0.166 
Sample size 529 342 


SOURCE: MDRC calculations based on responses to the PACE evaluation 12-month follow-up survey. 


NOTES: Outcomes refer to the period from study enrollment to survey interview date, unless otherwise noted. 

Results in this table are regression-adjusted, controlling for pre-random assignment characteristics. Statistical 
significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

aBinge drinking is defined as having five or more drinks within a couple of hours. 

‘Girls were asked about use of cocaine, inhalants (Such as glue or solvents), and pills without a doctor's 
prescription. 


°Measures refer only to voluntary sexual activity in the past three months. 


In the 12-month follow-up survey, sample members were also asked about their involve- 
ment in delinquent property and violent incidents in the past three months. These findings are 
self-reported and do not necessarily reflect any formal involvement in the juvenile justice system, 
which is discussed in the next section. Some research suggests that self-reported delinquency is 
predictive of future juvenile justice involvement.”° 


Types of self-reported property delinquency in the survey included shoplifting, using 
someone’s car without permission, stealing, breaking something on purpose, driving a car 
without its owner’s permission, and using someone else’s credit card or bank card without 
the person’s permission or knowledge. Self-reported violent incidents referred to physical 
fights or using or threatening to use a weapon to get something from someone. As shown in 


Farrington et al. (2003). 
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Table 3.4, the program group was less likely to have reported involvement in a property in- 
cident (40 percent) than the control group (50 percent), a statistically significant impact of 
nearly 10 percentage points. Among those who reported a property incident, the type of inci- 
dent most commonly reported was very general: “breaking something on purpose.” However, 
a sensitivity analysis (not shown) indicates that a statistically significant impact remains even 
when this type of property incident is excluded. Rates of involvement in violent incidents 
were much lower than for property incidents, and no statistically significant difference 
emerged between the two research groups. 


Table 3.4 also presents outcomes on risky sexual activity. Reported rates were low over- 
all, and no differences are seen in these outcomes across the two research groups. One year after 
random assignment, 9 percent of girls in the study were sexually active and did not use pregnancy 
prevention the last time they had a sexual encounter. Among high school girls in the study, 10 
percent reported this type of risky activity (not shown). By comparison, this rate was 4 percent 
among high school girls in Florida and 5 percent nationally.”! Additional details on sexual activity 
and other outcomes are included in Appendix Table G.1. 


Juvenile Justice System Involvement 


Sample members’ involvement in the juvenile justice system was measured across sev- 
eral outcomes, including whether they ever incurred a charge and the type of adjudication finding 
after a charge was incurred. Figure 3.1 shows select juvenile justice outcomes from the Florida 
Department of Juvenile Justice data for the 18-month period following study enrollment. Program 
and control group girls incurred one or more charges at nearly identical rates (22 percent of the 
program group and 21 percent of the control group) during the 18-month follow-up period. In- 
curring a charge is comparable to an arrest in the adult criminal justice system, and being “‘adju- 
dicated delinquent” is similar to a conviction. 


These rates are unsurprisingly higher than the rate for the broader Florida female 
youth population.” This finding speaks to the target population that PACE serves and their 
particular set of risk factors, including prior justice system involvement for some (28 percent 
of the study sample). Notably, the portion of the research sample involved in the juvenile 
justice system in the 18 months after study enrollment is similar to the percentage of those 
who had ever been involved before study enrollment. Over half the sample members who 
incurred a charge within the 18-month follow-up period had prior juvenile justice involve- 
ment at the time of study enrollment. 


Looking at the pattern of charges more closely, most girls who incurred a charge during 
the follow-up period did so for the first time shortly after random assignment. These findings 


*1Kann et al. (2016). The question applied to a girl’s last sexual encounter. This information was not avail- 
able for middle school girls. 


?2One percent of all girls in Florida were arrested in 2014 (Florida Department of Juvenile Justice 2017). 
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Figure 3.1 
Impacts on Juvenile Justice Outcomes 


Difference 
22.0% 
21.2% 0.8 


5.5% 
Adjudicated delinquent 12 
4.3% 
9.7% -1.5 
Adjudication withheld 
11.2% 
14.7% 
Adjudicated not delinquent 1.1 
13.7% 


0.1% 0.2 
0.0% 


| Program group ea Control group 


SOURCE: MDRC calculations based on data provided by the Florida Department of Juvenile Justice. 


Ever incurred a charge 


Transferred to adult court 


NOTES: Sample size is 673 for the program group and 452 for the control group. 
Results in this figure are regression-adjusted, controlling for pre-random assignment characteristics. No 
statistically significant differences were found between the program and control groups on any of these outcomes. 
Rounding may cause slight discrepancies in sums and differences. 
One sample member included in the adjudicated not delinquent category did not have any resolved cases. 


support the idea that many girls applied to PACE in a moment of crisis, and that they stabilized 
shortly after. This analysis does not consider whether sample members continued to incur charges 
after their first charge, post-random assignment. As shown in Figure 3.2, about 12 percent of girls 
(across the full study sample in both research groups) incurred a charge in the first six months 
following study enrollment. In the 6- to 12-month period following study enrollment, the portion 
of girls who incurred their first charge since enrollment drops to 6 to 7 percent; in the 12- to 18- 
month period following study enrollment, the rate drops further (about 3 percent for both research 


groups). 


Several other impacts on outcomes related to delinquent adjudications, charge types, and 
civil citations are shown in Appendix Table G.2. The rates of these outcomes were very low for 
both the program and control groups, and there was no pattern of statistically significant differ- 
ences between groups on these outcomes. 


Figure 3.2 


Timing of First Charge Incurred Since Random Assignment, Full Sample 


Time since 
random assignment 


0 to 6 months 12.1% 


6 to 12 months 6.6% 


12 to 18 months 


SOURCE: MDRC calculations based on data provided by the Florida Department of Juvenile Justice. 


Further follow-up with the study sample might be necessary to see whether differences 
emerge between the two groups with respect to justice system involvement. The sample members 
are still relatively young and just reaching the age when criminal behavior may start to emerge.” 


Positive Outlook and Future Orientation 


Through its holistic programming, PACE aims to help girls advocate for themselves and prepare 
for their futures. This section presents impacts on outcomes related to hope and future outlook, 
based on responses to the 12-month follow-up survey. Overall, girls felt positive about their fu- 
tures and meeting their goals. On most measures presented in Table 3.5, girls in the two research 
groups look fairly similar. 


The 12-month follow-up survey included a six-question scale to assess girls’ agency and 
perceived ability to work toward goals.** The statistically significant effect on positive outlook, 
which amounts to an effect size of 0.20 standard deviations, indicates that PACE caused an im- 
provement in girls’ positive outlook, as measured by this scale. Positive outlook, or feelings of 
hope, may lead to self-esteem.** Hope can be defined through the lens of goal setting: identifying 
goals and believing in one’s ability to take the steps to reach them.”° 


3National Institute of Justice (2014). 
24Snyder et al. (1997) 
5Snyder et al. (1997). 
6Snyder et al. (1997). 
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Table 3.5 
One-Year Impacts on Hope and Future Orientation 


Program Control Difference 


Outcome Group Group (Impact) P-Value 
Positive outlook score’ (average) 3.4 3.3 0.1 *** 0.003 
Positive coping score” (average) 3.5 3.5 0.0 0.893 
On track to get high school diploma or GED certificate° (%) 93.4 91.1 2.3 0.207 


Perception of meeting education goal (%) 


Thinks she will reach goal 64.5 64.7 -0.3 0.940 

Thinks she will exceed goal 10.4 8.1 2.3 0.268 

Does not think she will meet goal 25.1 27.2 -2.0 0.510 
Has strong idea of career she wants to pursue (%) 64.0 62.6 1.4 0.681 
Sample size 526 341 


SOURCE: MDRC calculations based on responses to the PACE evaluation 12-month follow-up survey. 


NOTES: Results in this table are regression-adjusted, controlling for pre-random assignment characteristics. 
Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

aScore is calculated as an average of responses to six items on the Children's Hope Scale. The possible score 
range is 1 to 4. The standard error is 0.036 and the effect size is 0.202. 

Score is calculated as an average of responses to a subset of items from the Youth Survey in the Urban 
Corps Assessment Package. The possible score range is 1 to 4. The standard error is 0.042 and the effect size is 
0.009. 

°Measure is nonexperimental and only includes sample members who did not have a high school diploma or 
General Educational Development (GED) certificate at the time of survey completion. 


As shown in Table 3.5, the majority of girls — more than two-thirds — in both research 
groups thought they would meet or exceed their education goals at the time of the survey. Similar 
percentages of girls reported that they had a strong idea of the career they wanted to pursue (64 
percent of the program group and 63 percent of the control group). There were no statistically 
significant differences between the program and control groups on these outcomes. Having a clear 
vision of one’s future self indicates an ability to set goals and undertake the planning necessary 
to meet them. Females tend to have stronger future orientation, higher instances of planning, and 
fewer fatalistic beliefs than males.’ 


Box 3.2 provides a more detailed look at the perspectives of sample members and parents 
or guardians on the girls’ futures, based on information collected in semistructured telephone 
interviews. 


27Johnson, Blum, and Cheng (2014). 
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Box 3.2 
Girls’ and Parents’ Perspectives on Future Goals 


Interviews with girls in both the program and control groups indicated that they had a hopeful 
outlook about the future and reaching their goals. They were introspective about themselves and 
their place in the world and were focused on reaching specific steps in their education and/or 
careers. 


¢ Most girls indicated a desire to attend college and were motivated to pursue a career. In 
particular, nursing was of interest to girls interviewed, and girls were for the most part 
clear-eyed about the educational requirements for that career path. 


Though girls were optimistic about their goals and the steps it would take to achieve them, 
they were also often realistic about their challenges. As one girl expressed it, “There are 
some things in life you just have to deal with; my depression and anxiety aren’t going to 
disappear. I have to learn to cope with it.” 


Parents and guardians echoed the sentiments expressed by the girls: 


* Most hoped that their daughters would go to college, similarly voicing an interest in the 
medical field. 


Although some parents clearly noted the obstacles that might prevent their daughters from 
going to college, such as lack of motivation or inadequate grades, most seemed optimistic 
that it would remain a possibility. One parent noted, ““She’s got a lot of potential; she could 
do anything she wants if she uses her talents to her advantage.” 


SOURCE: MDRC telephone interviews with study participants and their parents or guardians. 


Subgroups 


Appendix Tables G.3 through G.6 present exploratory subgroup analyses to assess whether 
PACE had different impacts among subgroups of the study sample, as defined by certain baseline 
characteristics. Four subgroups — age, prior juvenile justice involvement, prior school discipline, 
and a history of being held back in school — were chosen, largely based on conversations with 
PACE program staff members, who suggested that these characteristics in particular may influ- 
ence how girls approach or benefit from the program.” As shown in the tables, however, there 
are no differences between impacts on selected key outcomes for any of the subgroups. 


Again, it is important to note that these were exploratory analyses, which can inform 
future research for this population. This evaluation did not have sufficient statistical power to 
detect differences between subgroups. 


8Race/ethnicity was not chosen as a subgroup because it varied significantly by site (see Appendix Figure 
G.1); therefore, any differences that emerged between impacts could be related to site characteristics and/or 
race/ethnicity. 
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Conclusion 


PACE led to improvements in academic outcomes in the first year after study enrollment. Spe- 
cifically, program group girls were more likely to be present in school for more days, and high 
school girls in the program group were more likely to be on track, according to a composite meas- 
ure. In the first year after study enrollment, girls in both research groups were hopeful about their 
futures, expected to meet their goals, maintained positive relationships with family members and 
friends, and engaged in low levels of risky behavior. These characteristics differ considerably 
from the crisis state experienced by many girls at the time they enrolled in the study. There were 
no statistically significant differences on juvenile justice outcomes; any impacts on formal juve- 
nile justice involvement are expected to take longer to emerge. 


The following chapter will discuss the cost implications of these findings. 
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Chapter 4 
Analysis of PACE Costs and Net Costs 


This chapter examines the costs involved in providing PACE services and compares them with 
the costs of services available to the control group. First, the gross costs of PACE indicate what 
resources are required to operate a PACE center and the cost of serving the average girl enrolled 
in PACE. Some of these costs (for example, for academic instruction) would be incurred even in 
the absence of PACE; therefore, the net cost represents the additional cost of providing PACE 
beyond the cost of what the control group received. The chapter then compares the cost of PACE 
with the cost of other youth programs. Finally, because a year is too short a time frame to expect 
to fully recoup the costs of an intensive program like PACE, the chapter concludes with a discus- 
sion of the impacts PACE would need to produce to be cost effective in the long term. 


In sum, the average cost of serving a girl at PACE is about $10,400 more than the cost of 
serving a girl through the services she would receive in the absence of PACE. The analysis finds 
that the cost of providing academic services at PACE is comparable to the cost of public school. 
The greater investment required by PACE is driven primarily by the intensive social service sup- 
port the program provides, which is rare in the public schools many girls in the control group 
attended. The analysis found that if PACE were able to improve girls’ graduation rates in the long 
term, the program would ultimately pay for itself. 


Estimating the Cost of PACE 


PACE provides a staffing ratio and level of social services not typically found in traditional 
schools, which many of the girls in the study attended before they entered the study, and where 
many of the control group girls remained after random assignment. The more intimate and inten- 
sive nature of services like those PACE provides, which may be key factors in its positive impacts 
on academic engagement as presented in Chapter 3, come with higher costs. Table 4.1 presents 
costs for an average center. (All costs are expressed in 2017 dollars.) It costs about $1,820,000 
annually to run an average size PACE center, a sum which includes a portion of PACE headquar- 
ters costs. Based on the average length of stay of girls in the study (eight months), this translates 
to an average cost per girl served of about $23,500. Appendix H provides details about how the 
cost analysis was conducted. 


Table 4.1 further breaks down costs into functional service categories, to provide infor- 
mation on the relative costs of each component of PACE. Costs are attributed to the following 
components: 
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Table 4.1 
PACE Program Costs, Per Center and Per Girl 


Service Component _—sAnnual Cost Per Center ($) Cost Per Girl ($) Percentage of Total 
Outreach and recruitment 112,116 1,447 6.2 
Academics 848,425 10,952 46.6 
Social services 806,544 10,411 44.3 
Follow-up services 53,321 688 2.9 


Total cost 1,820,406 23,499 100 


SOURCES: MDRC calculations based on information provided by PACE and the PACE management 
information system. 


NOTES: Annual cost per center assumes sufficient space and staff for 51 girls to be served at one time. 
Cost per girl assumes an average length of stay at PACE of 7.9 months. 
All unit prices have been adjusted for inflation using the rate provided by the Bureau of Labor Statistics 
Consumer Price Index (CPI) inflation calculator (inflation rate = 1.03). It was assumed that the 2015 prices 
were listed at the June 2015 rate. These prices were converted to May 2017 prices. 


e Outreach and recruitment: promoting PACE in the community and screening 
applicants for eligibility.! 


e Academics: providing the academic program, including management, instruc- 
tion, academic advising and assessment, life skills classes, and books. 


e Social services: management and delivery of social service programs, includ- 
ing case planning and counseling, individual and group therapy, parental en- 
gagement, and an incentive program. The category also includes supportive 
services such as transportation, lunches for girls, and PACE T-shirts. 


e Follow-up services: managing and providing follow-up services to girls, pri- 
marily the transition counselor’s salary. 


Roughly half of PACE’s spending is for social services and half for the academic pro- 
gram. This aligns with PACE’s service model, which puts a priority on helping girls with rela- 
tional, health, and other nonacademic issues along with efforts to get her back on track academi- 
cally. The minimal spending on follow-up services echoes the implementation research finding 
that this part of the model was not a major program element at most centers.” 


‘During the period of study enrollment, PACE increased the staffing for this function in order to manage 
the increased volume of applicants and study-related paperwork. These study-related staffing costs have been 
excluded from the cost estimates presented here. 


?Treskon, Millenky, and Freedman (2017). 
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PACE Funding Sources 


Figure 4.1 shows that most of PACE’s funding comes from public sources.? PACE’s two 
largest sources of funding are local school districts, which provide funding for academic services, 
and the Florida Department of Juvenile Justice (DJJ), which supports its social services. The 
school district provides nearly $5,000 per girl, which is about half what it costs PACE to provide 
academic services. DJJ provides about $11,000 per girl through a funding stream directed toward 
prevention services. Through local and headquarters-led fundraising efforts, the organization 
raises about 18 percent of its budget from private sources, including individuals, corporations, 
and foundations. 


Figure 4.1 


PACE Revenue by Category 
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SOURCE: MDRC calculations based on information provided by PACE. 


NOTE: Percentages may not sum to 100 because of rounding. 


3These calculations were done at the organization-wide level. 
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Estimating the Net Costs of PACE 


While the program group girls were primarily participating in PACE, the girls in the control group 
were using other resources available in their community. The difference between these two sets 
of services is what drove the impacts the study found. To determine the cost to society of these 
impacts,* the net cost analysis examines the difference between the cost per average program 
group girl and the cost per average control group girl. This is called the net cost of the program. 


Unit Costs 


The starting point for the net cost analysis is the unit cost — the cost of providing a par- 
ticular service to a girl. The specific services examined are those the program group and control 
group girls received during the 12 months following their enrollment in the study — educational 
services (PACE or public school) and professionally delivered social services (within PACE or 
in the community). 


Table 4.2 presents the unit costs (cost per girl) used in the analysis for two types of ser- 
vices. The top panel shows the unit costs of PACE services, which come from the PACE program 
component costs presented in Table 4.1.° The bottom panel shows unit costs for services provided 
outside of PACE, which were mainly received by the control group. The largest cost in this sec- 
tion is for public school. This analysis assumes that girls who participated in a school other than 
PACE did so at a public school for a one-year period.*® The costs of public school were estimated 
from Florida Department of Education financial data.’ Since these prices are averages across all 
students, they probably reflect a low estimate for control group girls, since given their risk factors, 


4This analysis focuses on the societal costs of PACE and control group services. Though it is typical in cost 
analysis to assume participant opportunity costs (such as the price of missed wages while participating in a train- 
ing program), this analysis does not assume such costs, because the study participants are mostly of school age. 
Nor does the analysis include opportunity costs of parents participating in PACE’s parental engagement compo- 
nent, because meetings at PACE occur infrequently (typically two times per month), and staff members strive to 
hold these meetings at a place and time that is convenient for the parents. 

5PACE’s annual costs have been adjusted to 7.9 months to reflect the average length of stay at PACE. 
Though PACE girls commonly go back to public school after they leave PACE, public school costs for the 
remaining months of the 12-month period are not included in the program group estimates. The available data 
do not allow for estimates of what proportion of the remaining four months of the year the program group girls 
spent in public school. Transfers from PACE to public school could have happened right away, or they could 
have taken several months, particularly if the girl made the transition during summer break. On the high end, 
public school costs could add as much as $3,160 to a program group girl’s average costs if she participated in 
public school the entire four months she was not at PACE, or as low as $0 if she did not participate in public 
school at all that year. 

°Though control group members on average did not attend school year-round, yearlong costs are appropriate 
since many costs to the school district for a year are fixed. 

’The unit costs (or costs per student) are calculated from annual expenditures per unweighted full-time 
equivalent (UFTE) staff member and include general, special revenue, debt service, and capital projects funds. 
The UFTE was averaged among the six counties that correspond to the six PACE centers selected for the cost 
analysis (Florida Department of Education 2015a). 
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Table 4.2 


Unit Cost Per Service Accessed by Sample Members 


Service Component Cost Per Girl ($) 
PACE costs 

Outreach and recruitment 1,447 
Academics 10,952 

Social services 10,411 

Follow-up services 688 


Non-PACE costs 


Academics (public school) 10,500 
Counseling and therapy 2,640 
Case management 815 


SOURCES: MDRC calculations based on data from the Florida Department of 
Education, Florida Medicaid Billing Rates, and PACE. 


NOTES: All unit prices have been adjusted for inflation using the rate provided by the 
Bureau of Labor Statistics Consumer Price Index (CPI) inflation calculator (inflation rate 
= 1.03). It was assumed that the 2015 prices were listed at the June 2015 rate. These 
prices were converted to May 2017 prices. 


they are more likely to need specialized resources that may be offered by the school, such as 
counseling or academic support. However, the availability and intensity of social services in pub- 
lic schools are typically much less than what PACE offers to its participants. 


Girls in both research groups reported that they participated in counseling and case man- 
agement services outside of PACE or public school. The unit costs for services outside of PACE 
or public school were taken from Florida Medicaid billing rates for individual and family therapy 
and targeted case management, and the prices assume participation at the frequencies reported by 
girls in the survey.® 


Adjusting Unit Costs to Reflect Participation 


Since not all sample members participated in services, applying the full unit cost to every 
sample member would provide an overestimate of the costs of services for that group. The unit 
costs are therefore adjusted to reflect actual participation rates in those services for each research 
group. 


Agency for Health Care Administration (2017a, 2017b). This analysis assumes that girls who received the 
services outside of PACE participated in counseling for one hour, three times per month, for a year and partici- 
pated in case management two times a year. 
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The top panel of Table 4.3 shows the participation rates used for each research group. 
This information is drawn from the PACE management information system, Florida Department 
of Education data, and PACE survey data. Since both groups in the sample went through PACE’s 
outreach and recruitment process before random assignment, all sample members received those 
services. For participation in PACE services beyond recruitment, the calculations use the percent- 
age of program group girls who enrolled in PACE.’ A small portion of the program group — 
about 4 percent — did not enroll in PACE and are therefore allocated public school costs and no 
PACE costs. Public school costs are allocated to the nearly 90 percent of the control group who 
were enrolled in an academic program at some point during the school year following random 
assignment, according to the Florida Department of Education records. '° 


Participation rates for services outside of PACE or public school come from self-reported 
data from the follow-up survey of girls. Girls were asked whether they had received counseling 
or therapy since random assignment, and the source and frequency of the service. They were also 
asked whether they received case management services, such as transportation assistance, meals, 
or housing. The participation rates in the table reflect only services received from a professional 
source. For the program group, the participation rates for counseling and therapy and case man- 
agement reflect the portion of girls who reported receiving the services from a professional source 
other than PACE. 


The bottom panel of Table 4.3 shows the adjusted costs per sample member. This is cal- 
culated by multiplying the unit cost by the participation rate for each component. 


Calculating Net Costs 


Finally, the costs of serving the control group are subtracted from the costs of serving the 
program group, yielding the net cost estimates. The bottom line in Table 4.3 shows that the cost 
of serving a girl in the program group was about $10,400 more than the cost of serving a girl in 
the control group. 


Figure 4.2 illustrates the differences in the costs of the services received by the two re- 
search groups. Academic costs were similar overall ($10,268 for the program group compared 
with $9,408 for the control group); social services accounted for most of the difference in costs. 
The program group received nearly $9,000 more in social services, mostly from PACE. This is 
not surprising given PACE’s focus on social services. When only social services outside of PACE 
are examined, the control group’s costs are about $500 higher than the program group’s, mainly 
driven by their higher participation in counseling or therapy. Given the limited availability of 


*Participation rates for PACE are drawn from the full study sample, not just the six centers used to generate 
the cost estimates. 

‘OThe research team conducted a sensitivity analysis on the types of schools attended by the control group 
and estimated how that affected the net costs. For example, academic costs comparable to PACE were estimated 
for the portion of the control group that reported attending alternative schools. Differences in the net costs were 
small (under $100). 
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Table 4.3 


Estimated Gross and Net Costs Per Sample Member 


Program Control Difference 
Service Component Group Group (Net) 
Participation rate (%) 
Outreach and recruitment (PACE) 100.0 100.0 0.0 
Academic services (PACE) 90.3 0.0 90.3 
Academic services (public school) 3.6 89.6 (86.0) 
Social services (PACE) 90.3 0.0 90.3 
Follow-up services (PACE) 90.3 0.0 90.3 
Counseling and therapy (non-PACE) 22.0 40.9 (18.8) 
Case management (non-PACE) 12.9 12.5 0.4 
Cost per sample member ($) 
Outreach and recruitment (PACE) 1,447 1,447 0 
Academic services (PACE) 9,890 0 9,890 
Academic services (public school) 378 9,408 (9,030) 
Social services (PACE) 9,401 0 9,401 
Follow-up services (PACE) 622 0 622 
Counseling and therapy (non-PACE) 582 1,079 (497) 
Case management (non-PACE) 105 102 3 
Total cost per sample member per year ($) 22,425 12,037 10,388 


SOURCES: MDRC calculations based on data from the Florida Department of Education, the PACE 
evaluation 12-month follow-up survey, the PACE management information system, and PACE. 


NOTES: All unit prices have been adjusted for inflation using the rate provided by the Bureau of Labor 
Statistics Consumer Price Index (CPI) inflation calculator (inflation rate = 1.03). It was assumed that the 2015 
prices were listed at the June 2015 rate. These prices were converted to May 2017 prices. 

Rounding may cause slight discrepancies in sums and differences. 


counseling and therapy at public schools, it makes sense that control group members used these 
services in the community at higher rates, whereas the program group was able to receive them 
from PACE. The cost of follow-up services for the control group could not be estimated because 
of limited data. However, as discussed in Chapter 2, the control group received follow-up services 
at low rates, so these services would not be likely to have a large effect on the overall net cost 
estimate. 
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Figure 4.2 
Estimated Costs Per Sample Member, by Group 


Program group 


Outreach and 


recruitment $1,447 


Academic services $10,268 


Social services $10,089 


Follow-up services $622 


Control group 


Outreach and 


recruitment $1,447 


Academic services $9,408 


Social services $1,181 
Follow-up services? — 


SOURCES: MDRC calculations based on data from the Florida Department of Education, the PACE evaluation 12- 
month follow-up survey, the PACE management information system, and PACE. 


NOTES: All unit prices have been adjusted for inflation using the rate provided by the Bureau of Labor Statistics 
Consumer Price Index (CPI) inflation calculator (inflation rate = 1.03). It was assumed that the 2015 prices were 
listed at the June 2015 rate. These prices were converted to prices from May 2017, the most recent date with 
available data. There may be rounding differences between the CPI inflation calculator output and the prices 
presented in this figure. 

Rounding may cause slight discrepancies in sums and differences. 

@The cost of follow-up services for the control group could not be estimated because of limited data, but the 
control group received these services at low rates. 


PACE Costs Compared with the Costs of Other Programs 


In this section we compare the costs of PACE with other services available to children who are 
behind academically or at risk of delinquency. PACE costs more than attending regular public 
school, but as shown in Chapter 2, girls who attended PACE received more services than girls in 
the control group. Girls served by PACE were often coming from a more traditional school and 
looking for success in a different environment. Related to the more intense services received by 
girls who attended PACE, the program’s costs should be examined in the context of the positive 
impact this evaluation found on some measures of academic engagement. 
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PACE’s program model is unusual, both in Florida and in the United States. There are 
few nonresidential programs for at-risk children that offer a similar level of social support along- 
side small academic classes, and there are no comparable gender-responsive programs operating 
in Florida. Comparing the costs of PACE with those of other alternative schools would be helpful, 
but there is very little information available about the costs of schools that share many of its main 
components. This section compares PACE with the costs of public school, residential programs, 
and programs for disconnected youth. 


PACE’s small center size contributes to its costs. The average center serves about 50 
girls, which is much smaller than the traditional secondary school in Florida. The state has some 
of the largest high schools in the nation.!! Girls and parents of girls who attended PACE said 
PACE’s size was part of the attraction of the school. There was the perception that the girls’ 
challenges would get more attention than they might in a large public school, and parents saw 
PACE as a safer environment for their children. However, PACE centers cannot take advantage 
of the economies of scale available to larger schools. 


Therefore, the difference between PACE costs and those of a regular public school are 
primarily due to staffing. Compared with public schools in Florida, PACE has smaller class sizes 
and lower ratios of girls to social service staff. Florida has a statewide ratio of 16 students per 
teacher. '* PACE teachers reported a ratio of 11 students per teacher.'* The state of Florida does 
not require that schools have counselors, and according to one estimate, Florida schools have one 
counselor for every 485 students. '* In comparison, the average-size PACE center had three coun- 
selors for 51 girls (for a ratio of one counselor for 17 girls).'> When all PACE staff members, 
including administrative support personnel, are included in the calculations, PACE’s staff-to- 
student ratio is considerably smaller than that of Florida’s public schools. PACE centers in the 
study had an average of one staff person for every three girls.'° Florida Department of Education 
reports an average of one full-time staff person for every eight students. (Its calculation includes 
the service staff in the calculation, while PACE’s calculation does not, since custodial services 
are contracted out and are included in overhead costs.) !” 


‘Available data from 2009 indicates that the average secondary school size in Florida was around 1,700 
students (Florida Department of Education 2009). 

Florida Department of Education (2015a). The ratio is calculated by dividing student membership by 
teachers at traditional schools. This includes elementary schools; separate estimates for middle and high schools 
were not available. 

>Treskon, Millenky, and Freedman (2017). 

4National Association for College Admission Counseling and American School Counselor Association (n.d.). 


>Caseload size was reported by counselors in the PACE staff survey. School counselors and PACE coun- 
selors play different roles. Though school guidance counselors may play a role in tracking student progress and 
referring students to resources to support their goals, PACE counselors have a much greater focus on the social 
and emotional needs of girls on their caseload (Treskon, Millenky, and Freedman 2017). 

‘6Treskon, Millenky, and Freedman (2017). 

Florida Department of Education (2015b). In addition, a small number of staff members at each PACE 
center work part time, so an exact comparison of PACE’s staff-to-girl ratio to the Florida Department of Educa- 
tion’s report of full-time equivalent staff is not practicable. 
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PACE costs less than residential programs or confinement. A Florida estimate of non- 
secure residential confinement for juveniles was $47,691 per year (a comparable eight-month 
period would be $31,794 compared with PACE’s $23,701).!’ PACE costs are similar to those 
of the comprehensive programs for young people who are disconnected from school and work, 
such as YouthBuild and Job Corps.’ All these programs have a secondary-education comple- 
tion component but include other elements that contribute to their costs, such as stipends for 
participation, vocational training, or a residential component. PACE serves a younger popula- 
tion than these programs do, and, notably, all these programs are focused on high school com- 
pletion or the equivalent, while PACE is focused more on academic progress. YouthBuild, the 
most expensive of the three, is delivered through small programs similar in size to PACE, a 
factor that contributes to its costs.”° 


The Potential Economic Benefits of PACE 


The net cost estimate reflects the investment society is making in girls who attend PACE 
above and beyond what they would have otherwise spent on the girls by serving them in a tradi- 
tional public school environment. Most of these costs are concentrated in a short period of time 
— about eight months for the average participant — and many of the impacts presented in Chap- 
ter 3 cannot easily be translated into economic benefits in the short term. Ultimately, however, 
the aim of PACE is to change the long-term trajectory of its participants by getting them back on 
track academically and promoting prosocial behaviors. 


Improving high school graduation rates has the potential to generate large economic ben- 
efits. Compared with dropouts, high school graduates have been shown to make larger economic 
contributions to society by having lower unemployment rates, having higher earnings, paying 
more taxes, and relying less on public assistance. One estimate found that female high school 
graduates make nearly $300,000 more over the course of their lifetimes than those who do not 
complete high school.”' Adjusting those earnings to net present value (since future earning must 
be discounted to account for the fact that a dollar in the future is worth less today) yields an added 
lifetime value for a high school diploma of $140,000.” If the sizable difference in the “on track” 
measure reported in Chapter 3 endures, and PACE increases high school graduation rates by at 
least 8 percentage points, PACE would be cost effective in the long term.” 


'8Justice Policy Institute (2014). 


‘Estimated costs for YouthBuild are $24,521 per participant; for JobCorps, just under $24,500 per partici- 
pant (Miller et al. 2018). 


°Wiegand et al. (2015). 
?!Tamborini, Kim, and Sakamoto (2016). 


?2Tamborini, Kim, and Sakamoto (2016). Net present value is calculated based on education level at age 20 
and assumes a 4 percent discount rate. 


3This calculation assumes that the net increase in cost to provide services at PACE versus business as usual 
is $10,388 per girl (or $1,038,800 for 100 girls). Eight additional graduates would result in $1,120,000 more in 
benefits over the course of the girls’ lifetimes. 
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That estimate considers only improved high school graduation rates, but such an impact 
would probably go hand in hand with other benefits. Girls who are successful at school are less 
likely to be involved in the juvenile justice system, and high school graduates are less likely to be 
involved in the criminal justice system as adults. Long-term impacts on these outcomes would 
generate additional long-term cost savings. Of course, further follow-up with the study sample 
would be necessary to calculate any longer-term differences in academic or other outcomes. 
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Chapter 5 
PACE Today and Study Implications 


PACE Today 


This evaluation — and the promising academic outcomes it presents — reflects the services pro- 
vided from mid-August 2013 through 2015. PACE is a growing and evolving organization, and 
more centers have opened across the state since the evaluation began in 2013. PACE has a strong 
central office, or headquarters, to support the centers’ implementation of the program services 
and advocate for the population it serves, and its leaders continue to assess those services as well 
as their implementation. Over the course of the evaluation period, PACE established a new man- 
agement structure at its central office, revised its policies on girls’ eligibility for the program, and 
revamped the life skills curriculum, among other changes undertaken. 


As preliminary implementation research findings became available, the leadership team 
at PACE worked actively to understand and address specific findings from the implementation 
research. As one senior PACE staff person said recently: “The MDRC evaluation has been a 
catalyst for change and reflection.” For example, the study team found no standard set of recom- 
mended approaches for counselors to use in sessions with girls. As a result, PACE developed a 
more defined toolbox for counselors and therapists to use during sessions. Similarly, a synthesis 
of the classroom observation data showed room for improvement in the way teachers provided 
academic support in the classroom. PACE is addressing this by giving teachers more training and 
support on instructional methods and shifting away from independent work in the classroom. A 
newly developed tool allows managers to assess teachers monthly and provide coaching. 


The terminology around PACE’s program culture is being addressed as well. Specifi- 
cally, the study team found that the program’s approach was implemented uniformly, but program 
staff members did not necessarily recognize that they were using gender-responsive or other spe- 
cific approaches to their work.! Partly to address this, PACE rolled out new in-person training 
sessions on the key pillars of PACE as a gender-responsive, strengths-based, and trauma- 
informed program. Role-specific training is also in development, to help program staff members 
more directly apply these elements to classrooms, counseling sessions, and other aspects of the 
program model. 


At the start of 2019, other initiatives are under way as well. As one executive director 
noted, there has been an increase in support for service delivery and a move to standardize deliv- 
ery across centers, while allowing for local differences. This support includes more specific tools 
and training for program staff members and managers to implement the program model’s com- 
ponents. Structurally, a new layer of management was added between the centers’ executive di- 
rectors and PACE’s headquarters. Newly appointed regional directors (who are center executive 


'Treskon, Millenky, and Freedman (2017). 
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directors themselves) provide more direct support and accountability for the other center execu- 
tive directors. In addition, PACE is focusing on enhancing its use of the data it collects, including 
implementing a new management information system (MIS). One goal of the new MIS is to better 
integrate data on PACE’s social services and academic services to give a more holistic view of 
the girls they are serving. The new MIS will also focus more on outcomes, as PACE aims to make 
its impact on the girls it serves easier to measure. 


In addition to opening new centers that offer the day program that is the focus of this 
evaluation, PACE has also been expanding one of its related programs, known as Reach. The 
program uses many of the same principles as the main PACE program but is focused on social 
services. Girls remain in their existing schools for academic services and receive PACE case 
management, counseling, and life skills class outside of school hours. These services may be 
offered at a PACE center or at a public school. As of September 2018, Reach operates in seven 
counties, with further expansion planned. 


Implications of the Study 


The evaluation concludes that PACE implements its unique, theory-based model well. This find- 
ing is discussed at length in the previous report, Helping Girls Get Back on Track.* The model’s 
gender-responsive approach and combination of academic and social services is not commonly 
found. In the absence of access to PACE, girls in the communities where PACE operates were 
not able to replicate the unique “PACE experience” through other services. 


Based on the random assignment study design, the evaluation also concludes that the 
PACE program positively affected girls in the short term, helping them to be more engaged and 
on track academically. Effects on other outcomes, including risky behavior and youth develop- 
ment, were not seen 12 to 18 months after enrollment. Further follow-up of the study sample 
would be necessary to see whether PACE affects longer-term academic and delinquency out- 
comes, among others. 


Across the full study sample a year later, girls reported some notable protective factors 
— that is, characteristics or conditions that may moderate the risk for negative outcomes. The 
girls, overall, had positive relationships with friends, family, and adults outside of the family. 
After their application to PACE at a time of relative crisis or a tipping point in their lives, things 
may have steadied for these girls, and they largely remain optimistic about their future. On some 
measures of risky behavior, such as substance use, one year later, the study sample exhibited 
lower rates than those of a broader population of girls in Florida and nationally. 


To be eligible for PACE, girls were required to have a parent or guardian present during 
the intake assessment and to show a willingness or a motivation to change things in their lives. 


?Treskon, Millenky, and Freedman (2017). 
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This combination may mean that girls in the study sample were particularly motivated and sup- 
ported by family, and that those assigned to the control group were thus likely to find other ser- 
vices and support that they needed in the absence of access to PACE. 


The services PACE provides for girls are more expensive than the less holistic alterna- 
tives available in their communities. However, the follow-up period of the study to date is too 
short to conduct a full benefit-cost analysis involving key outcomes such as high school gradua- 
tion and justice system involvement. Girls ranged in age from 12 to 18 at the end of the follow- 
up period, and the vast majority of the study sample were four to five years from reaching high 
school graduation. If the positive academic outcomes presented here, such as attendance and be- 
ing “on track,” lead to more girls advancing through grade levels on schedule and graduating 
from high school, then the program could be cost effective. 


The evaluation of the PACE Center for Girls adds to the growing literature regarding 
programs serving girls at risk of juvenile justice involvement and other problematic outcomes. It 
adds richly to the discussion about the implementation of gender-responsive programming, as 
presented in the 2017 report and companion brief.* The evaluation found that PACE reengages 
girls in academics in a way that could make a lasting difference, especially if it leads to higher 
rates of high school graduation and enrollment in postsecondary education. Engagement in school 
is also a protective factor against future justice system involvement. Longer-term follow-up 
would be necessary to investigate such potential benefits of PACE’s program; a full benefit-cost 
analysis would be more appropriate at that time. 


>Treskon, Millenky, and Freedman (2017); Treskon and Bright (2017). 
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Appendix A 


Site Selection, Random Assignment, 
and the Analysis Model 


Site Selection 


At the end of 2018, PACE operates 20 centers across Florida. Fourteen centers were included in 
the evaluation. Three centers opened after the evaluation began and so were not eligible to be 
included; two other centers were initially slated to be in the study but were excluded due to low 
enrollment during the study period; and one other center, located in a rural area, was excluded 
because of ethical concerns about a lack of other available services in the community for the 
control group. 


Random Assignment Process and Procedures 


Study enrollment was divided into two start-up phases, with eight centers starting random assign- 
ment in August 2013 and the other six in early 2014. Girls applying to PACE continued to enroll 
in the study through early November 2015. 


The study team customized a manual for PACE staff members outlining random assign- 
ment and other research procedures. The manual detailed the research design and the steps re- 
quired of program staff members at each step from outreach through enrollment. The study team 
visited each center to train program staff members in these procedures, including the procedures 
for entering data into the MDRC random assignment system. Once random assignment began, 
the study team was in regular communication with program staff members to answer any ques- 
tions and monitor study enrollment. 


PACE’s standard intake and eligibility determination process was conducted before ran- 
dom assignment. Intake appointments completed at the PACE center were the first step in this 
process. An appointment included an introduction to PACE, an orientation to the evaluation, 
completion of study paperwork (consent forms and contact information), and a formal assessment 
to allow staff members to understand the girl’s background and risk factors. After an intake ap- 
pointment, a group of staff members at the center discussed the assessment’s findings and made 
a final determination of eligibility. 


When a girl was deemed eligible for PACE, staff members enrolled the girl in the study 
by entering her into MDRC’s web-based random assignment system. In general, girls applying 
to the program were asked to take part in the study. However, some girls were not eligible for the 
study but were nonetheless invited to enroll in the program during the random assignment period. 
These included girls who had previously attended PACE, girls who had a family member already 
enrolled in PACE (whether in the study sample or not), and girls who were in state custody 
through the foster care system. It is difficult to determine exactly how many girls fell into each 
category, but the numbers did vary by site based on their recruitment strategies and referral part- 
ners. 


At times throughout the study enrollment period, pauses in random assignment were nec- 
essary because centers had difficulty meeting their set enrollment targets. During those times, all 
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eligible applicants were invited to attend the program, and those girls were not considered part of 
the study. The timing and length of such pauses depended on each center’s circumstances. All 
centers paused random assignment from mid-May to mid-July 2015, given anticipated low en- 
rollment over the summer months. During this two-month pause, there were 211 new PACE en- 
rollees across all participating sites. 


Because of study eligibility criteria and random assignment pauses, the study sample 
ranged from about 39 percent to 70 percent of all enrollees at participating sites. 


Analysis Model 


The basic estimation strategy is to compare average outcomes for the program and control groups. 
Regression adjustment in a linear regression model increases the power of the statistical tests. 


Outcome data were processed according to standard procedures to check for outliers or 
other irregularities. In the administrative records from the Florida Department of Education, a 
very small number of records had high values for days enrolled. For these cases, the days enrolled 
outcome was set to missing. In these cases, the outcomes for number of days present, number of 
days absent, and percentage of days absent, all of which depended on days enrolled, were also set 
to missing. 


The main impact analysis used the following model: 


Yy = ai + BPi + OX: + yy + €% 


where Yj is the outcome of interest (such as “suspended from school” or “engaged in delinquent 
behavior’) for sample member 7 in site /; 


P; is an indicator for membership in the program group; 


X; is the series of variables representing the baseline covariates for sample member i, including 
age, juvenile justice involvement, and race/ethnicity; 


o is the set of regression coefficients for Xj; 
y represents site fixed effects; 
and ¢; is the random error term for sample member 7 in site /. 


There were no observations missing any of the baseline covariates in list X;. The baseline 
covariates were limited to those in list _X; because rates of missing data on other baseline measures 
of interest differed for program group members compared with control group members. 


For the impact analysis using outcome data from the Florida Department of Education, 
X; includes only one baseline covariate, a categorical variable for age. These data were deidenti- 
fied before they were delivered to the study team and were linked only to this single baseline 
characteristic. 


66 


Appendix B 


Survey Response Bias Analysis 


The PACE evaluation 12-month follow-up survey collected information about PACE sample 
members across a range of domains, including service receipt, academic progress and employ- 
ment, outlook and youth development, social support and mental health, risky behavior, and sat- 
isfaction with the program. While the survey was fielded to the whole research sample, a portion 
of the sample did not respond. Because the survey results represent only a subset of the PACE 
sample, it is necessary to assess the reliability of any survey-based impacts in two ways. First, if 
girls who responded to the survey differ significantly from those who did not to respond to the 
survey, impacts may not be generalizable to the full research sample. Second, if program group 
members who responded to the survey differ significantly from control group members who re- 
sponded to the survey, estimates of the program’s effects using the survey could be biased. A 
third comparison is possible using administrative records. If the respondent and nonrespondent 
survey samples have similar impacts estimated using administrative data, it gives more credibility 
to the survey analysis. 


This appendix presents a description of the survey fielding efforts, assesses whether the 
impact estimates from the survey represent the program’s effects for the full study sample, and 
assesses the validity of the survey results for estimating program impacts. Overall, the analysis 
suggests that the results based on survey responses provide valid estimates of the program’s ef- 
fects and can be generalized to the full research sample with some caution. 


Survey Administration and Sample Characteristics 


The survey was fielded to the full research sample of 1,125 girls; 872 sample members responded 
to the survey (78 percent). The program group had a slightly higher response rate (79 percent) 
than the control group (76 percent). Fielding took place between October 2014 and November 
2016. Girls were contacted first via phone to complete the survey; if they could not be reached, a 
field representative of the survey firm followed up in person. 


Appendix Table B.1 shows selected characteristics for survey respondents and nonre- 
spondents, based on data collected by PACE at the time of study enrollment. Overall, there were 
few significant differences between the respondent sample and the nonrespondent sample. Non- 
respondents are expected to be in harder-to-reach groups, such as those with greater mobility, 
which is illustrated in the differences that emerged. At the time of study enrollment, nonrespond- 
ents were more likely to have skipped school, to have been involved with the juvenile justice 
system, and to have run away from home. 


These differences were also tested in a logistic regression model, in which the probability 
of response was regressed on the baseline characteristics shown in Appendix Table B.1. A test of 
joint significance indicated that when the baseline characteristics were taken as a whole, there 
was a Statistically significant difference between respondents and nonrespondents. 


The differences between the respondent and nonrespondent sample indicate that some 
caution should be exercised when generalizing the survey findings to the full research sample. 
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Appendix Table B.1 


Selected Baseline Characteristics of Respondents and Nonrespondents 


to the 12-Month Survey 


Characteristic 
Age 


Race/ethnicity (%) 
Black, non-Hispanic 
Hispanic* 

White, non-Hispanic 
Other 


School level at referral to PACE (%) 
Middle school? 


High school 


People participant lives with (%) 
Two parents 
Single parent 
Relative 
Other® 


Family income (%) 
$28,050 or below 
$28,051-$44,900 
Above $44,900 


Family has had case with the 
Florida Department of Children and Families (%) 


Skipped school at least 3 times in past 2 months (%) 
Failed 1 or more classes in past 6 months (%) 

Ever involved with the juvenile justice system (%) 
Ever sexually active (%) 

Ever run away from home (%) 


Sample size 


SOURCE: MDRC calculations based on data from the PACE management information system. 


NOTES: Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 


Respondents 


14.7 


46.4 
15.6 
37.3 

0.7 


22.7 
77.3 


35.3 
52.3 
10.2 

2.3 


41.4 
35.8 
22.8 


41.2 


33.0 


76.9 


24.4 


43.5 


25.3 


872 


Nonrespondents 


14.7 


39.9 
17.0 
41.9 

1.2 


19.8 
80.2 


33.3 
50.2 
11.9 

4.5 


42.0 
34.2 
23.8 


46.1 
40.9 
75.2 
36.8 
46.5 
36.0 


253 


aSample members are coded as Hispanic if they answered "yes" to Hispanic ethnicity. 
>This category includes sample members who were in fifth grade at the time of referral. 


"Other" includes nonrelative or foster care. 
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Full Sample 


14.7 


45.0 
15.9 
38.3 

0.8 


22.0 
78.0 


34.8 
51.8 
10.6 

2.8 


41.5 
35.5 
23.0 


42.3 
34.8 
76.5 
27.2 
44.2 
27.7 


1,125 


KK 


KK 


Comparisons Between Research Groups in the Survey 
Respondent Sample 


Although random assignment research designs ensure that the program and control groups are 
similar to each other at the time of study enrollment, there is a possibility that the selective 
nature of the survey response process could result in differences between the two groups of 
respondents. If differences emerge, then the impact estimates derived from the respondent sam- 
ple may be biased. 


Overall, program and control group respondents look nearly the same across baseline 
characteristics. Selected baseline characteristics for program and control group respondents are 
shown in Appendix Table B.2; there were no statistically significant baseline differences between 
the two groups. 


These differences were also tested in a logistic model, in which the probability of re- 
search group assignment was regressed on the baseline characteristics shown in Appendix Ta- 
ble B.2. A test of joint significance indicated that there was no statistically significant difference 
overall between the baseline characteristics of the program group respondents and the control 
group respondents. 


Consistency of Impacts 


While there were some differences in the baseline characteristics of respondents and nonrespond- 
ents, ultimately what is relevant is whether impact results for these two groups differ. This section 
compares impacts estimated from administrative records for the two groups. Impact estimates 
from administrative records provide the most reliable estimate of the program’s effects because 
they include the full research sample, not a potentially nonrandom subset of survey respondents. 
If the respondent and nonrespondent samples have similar administrative impact estimates, the 
survey analysis has more credibility, as there is less chance the survey impacts are biased. 


Appendix Table B.3 presents the results from this analysis, showing impacts on juvenile 
justice outcomes using records from the Florida Department of Juvenile Justice. (It was not pos- 
sible to conduct a similar test using records from the Florida Department of Education, as these 
data were deidentified and could not be linked to the survey sample.) Overall, the impacts on 
juvenile justice outcomes are nearly identical for the respondent and nonrespondent samples. A 
test of joint statistical significance across all juvenile justice outcomes further confirmed that im- 
pacts for the respondent and nonrespondent samples did not differ. 
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Appendix Table B.2 


Selected Baseline Characteristics of Program and Control Group 
Respondents to the 12-Month Survey 


Characteristic Program Group Control Group 
Age 14.6 14.8 
Race/ethnicity (%) 
Black, non-Hispanic 47.4 45.0 
Hispanic? 15.1 16.4 
White, non-Hispanic 36.8 38.0 
Other 0.8 0.6 


School level at referral to PACE (%) 
Middle school? 24.2 20.5 


High school 75.8 79.5 


People participant lives with (%) 


Two parents 35.3 35.2 
Single parent 53.5 50.3 
Relative 9.5 11.3 
Other® Af 3.1 


Family income (%) 


$28,050 or below 42.1 40.3 
$28,051-$44,900 36.4 34.8 
Above $44,900 21.5 24.8 


Family has had case with the 


Florida Department of Children and Families (%) 41.5 40.8 
Skipped school at least 3 times in past 2 months (%) 31.4 35.7 
Failed 1 or more classes in past 6 months (%) 76.0 78.3 
Ever involved with the juvenile justice system (%) 25.5 22.8 
Ever sexually active (%) 43.9 42.9 
Ever run away from home (%) 25.5 24.9 
Sample size 530 342 


SOURCE: MDRC calculations based on data from the PACE management information system. 


NOTES: No statistically significant differences were found between the program and control groups on any 
characteristics. 

aSample members are coded as Hispanic if they answered "yes" to Hispanic ethnicity. 

>This category includes sample members who were in fifth grade at the time of referral. 

“Other” includes nonrelative or foster care. 
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Appendix Table B.3 


Impacts on Juvenile Justice Outcomes at 18 Months, 
Based on Survey Response Status 


Respondent Nonrespondent 
Outcome (Impact in Percentage Points) Sample Sample 


Ever incurred a charge 0.6 2.4 


Adjudication type 


Adjudicated delinquent 1.4 1.9 
Adjudication withheld -2.2 -0.2 
Adjudicated not delinquent? 15 1.9 
Transferred to adult court 0.0 1.0 


Offense types for delinquent adjudication 


Person/violent -0.7 0.9 
Property 1.1 -0.6 
Drugs 0.2 -1.0 
Public order 0.3 -1.6 
Violation of probation or prior sanction 1.0 1.4 


Category for delinquent adjudication 


Felony 0.5 0.2 
Misdemeanor 0.7 -1.9 
Other? 1.0 1.4 


Disposition/sanction for delinquent adjudication 


Out-of-home placement 1.7 -0.5 
Community-based (including probation) 0.3 -0.6 
Other type of sanction® 0.5 1.9 
Sample size 872 253 


SOURCE: MDRC calculations based on data from the Florida Department of Juvenile Justice (DJJ). 


NOTES: No statistically significant differences on impacts were found between the respondent and 
nonrespondent samples. 

4aOne sample member included in this category did not have any resolved cases. 

"Other" is the DJJ categorization for violations of probation or prior sanction. 

°Other types of sanction include judicial warning, other judicial disposition, and disposition transferred to 
another county. 
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Appendix C 


Supplemental Baseline Tables 


Appendix Table C.1 


Characteristics at Baseline, by Research Group 


Characteristic (%) 


Age 
11-12 
13-14 
15-16 
17 


Race/ethnicity 
Black, non-Hispanic 
Hispanic? 
White, non-Hispanic 
Other 


School level at referral to PACE 
5th-6th grade 
7th-8th grade 
9th-10th grade 
11th-12th grade 


English is second language 


Has a learning disability 
Attention deficit disorder 
Dyslexia 
Other learning disability 


People participant lives with 
Two parents 
Single parent 
Relative 
Other? 


Family income 
$28,050 or below 
$28,051-$44,900 
Above $44,900 


Family has had case with the 
Florida Department of Children and Families 


Sample size 


77 


Full 
Sample 


8.4 
32.7 
49.2 

9.6 


45.0 
15.9 
38.3 

0.8 


8.8 
37.2 
45.2 

8.8 


2.0 


29.7 
19.5 
1.5 
8.9 


34.8 
51.8 
10.6 

2.8 


41.5 
35.5 
23.0 


42.3 


1,125 


Program 
Group 


8.8 
34.0 
47.5 

9.7 


44.4 
15.8 
39.1 

0.7 


9.7 
35.4 
45.9 

9.1 


1.9 


30.7 
21.0 
1.3 
8.8 


35.1 
52.7 
9.8 
2.4 


43.3 
35.0 
21.7 


43.1 


673 


Control 
Group 


8.0 
30.8 
51.8 

9.5 


45.8 
16.2 
37.2 


40.1 
44.3 


28.3 
17.3 


452 
(continued) 


Appendix Table C.1 (continued) 


SOURCE: MDRC calculations based on data from the PACE management information system. 


NOTES: No statistically significant differences were found between the program and control groups on any 
characteristics. 

aSample members are coded as Hispanic if they answered "yes" to Hispanic ethnicity. 

>"Other" includes nonrelative or foster care. 
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Appendix Table C.2 


Risk Factors at Baseline, by Research Group 


Characteristic (%) 


School engagement 
Recently expelled or suspended? 


Not currently enrolled in school 

Skipped school at least 3 times in past 2 months 
Had more than 15 absences in past 3 months 
Held back at least once 

Failed 1 or more classes in past 6 months 


Delinquency 
Ever incurred a charge” 


Ever charged with domestic violence 

Ever charged with burglary 

Ever charged with a drug crime 

Ever stolen from family, home, or neighbors 
Ever been on probation 

Currently on probation 


Has family member with criminal history® 


Has friends with delinquent record or who engage in 


delinquent behavior 
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Full 
Sample 


39.7 
26.7 
34.8 
41.7 
51.8 


76.5 


27.9 
6.0 
4.3 
3.5 

16.9 

12.7 

10.2 


64.2 


49.9 


Program 
Group 


39.4 
27.2 
32.9 
43.0 
51.2 


76.2 


28.4 
4.8 
4.5 
4.2 

17.1 

12.4 

10.1 


64.0 


49.2 


Control 
Group 


40.1 
25.9 
37.7 
39.7 
52.7 


77.0 


27.1 
7.8 * 


4.0 


16.6 
13.1 
10.4 


64.3 


51.1 


(continued) 


Appendix Table C.2 (continued) 


Full Program Control 
Characteristic (%) Sample Group Group 
Health and safety 
Currently using tobacco® 9.9 10.6 8.9 
Currently using drugs or alcohol® 14.9 15.0 14.6 
Ever sexually active 44.2 44.2 44.2 
Currently pregnant 1.4 1.9 0.7 * 
Ever run away from home 27.7 27.8 27.5 
Ever had thoughts about harming/killing herself 39.5 38.6 40.8 
Abused/neglected’ 38.2 38.4 37.9 
Neglected 9.0 9.4 8.4 
Physically abused 15.8 14.8 17.4 
Sexually abused 15.1 14.9 15.4 
Emotionally abused 21.8 21.5 22.4 
Sample size 1,125 673 452 


SOURCE: MDRC calculations based on data from the PACE management information system. 


NOTES: Certain characteristics listed here were captured in two different ways during the random assignment 
period, as noted below. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

aMeasure was defined as being currently expelled or suspended for approximately half of the sample and defined 
as one or more expulsions or suspensions in most recent school term for the other half. 

>In the juvenile justice system, people are not technically "arrested"; the terminology used is either "incurred a 
charge" or "referred." 

¢Refers to a criminal record (including imprisonment, probation, parole, and house arrest) for a parent, guardian, 
or sibling of the sample member for approximately half of the sample; for the other half, "family" includes other 
members of the household as well. 

dMeasure was defined as having used tobacco three or more times in the past 30 days for approximately half of 
the sample and defined as currently using tobacco for the other half. 

*Measure was defined as having used drugs or alcohol three or more times in the past 30 days for approximately 
half of the sample and defined as current drug or alcohol use for the other half. 

‘Measure includes a mix of documented and suspected instances of abuse or neglect. 
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Appendix D 


Procedures for Interviews with Parents and Girls 
and Qualitative Data Analysis 


Interview Process 


Between July 2015 and April 2016, MDRC conducted semistructured telephone interviews with 
52 girls and 40 parents of girls who enrolled in the program and control groups the prior year. 
The primary purpose of the phone interviews was to gain a clearer understanding of PACE ser- 
vices received or other resources available to girls in their communities. These interviews were 
structured around topics to highlight the control and program group service contrast. The parent 
interviews explore the extent to which parents participated in services provided to their daughters, 
either at PACE or elsewhere in the community. This section details the process of conducting 
these interviews. 


Interview Participant Recruitment 


Participants for in-depth interviews were recruited in the course of administrating the 12- 
month follow-up survey that was part of the impact analysis. That survey was fielded to program 
and control group members in person or by phone. As study participants were usually minors, the 
survey firm interviewer typically spoke with the parent or guardian first to explain the purpose of 
the survey and to schedule a time to complete it. 


During their contact with both parents and girls, interviewers from the survey firm in- 
quired about their interest in participating in another telephone interview at a later date. The sur- 
vey firm created a roster of interested girls and parents or guardians, which they sent to MDRC. 
MDRC then contacted the interested study participants to conduct the in-depth interviews. The 
parents and daughters who were interviewed did not necessarily match up. 


Across both program and control groups, girls who were interviewed were 16 years old 
on average and primarily lived with a single parent. Ninety percent of the parents interviewed 
were mothers, and most were employed at the time of the interview. The parents who were inter- 
viewed tended to have other children besides the girl in the study. 


Interview Procedures 


Girls and parents or guardians who expressed interest in participating in an interview 
were added to a roster on a rolling basis. The roster included relevant participant information, 
such as random assignment date and the best time to reach the individual. One interviewer was 
assigned to each participant. 


Interviewers contacted the candidates by telephone. All parents on the roster were con- 
tacted, and a subset from the roster of girls. If the candidate did not answer the call, the interviewer 
left a voicemail. In some cases, the participant returned the call to complete the interview, but in 
other cases the interviewer continued trying to reach the participant by phone. 


When the interviewer reached the candidate, the interviewers followed these procedures: 
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1. Obtain consent. For interviews with parents or guardians, interviewers used a consent 
script to obtain verbal consent from parents. Interviewers reminded the girls of their 
initial consent to participate in the study and obtained their consent to continue. The 
interviewer also asked for consent to digitally record the interview. 


2. Conduct interview. Interviewers used a protocol to guide the conversation. The goal 
of the interview was to gather information about the topics that felt important to the 
participants; it was less important to obtain answers to every question. Interviews 
lasted about 30 to 45 minutes. 


3. Obtain gift card information. At the end of the interview, the interviewer asked for 
information to send the participant a gift card in appreciation of his or her time. 


4. Conclude with thanks. The interviewer thanked the participant for sharing her 
thoughts. 


Protocols for Semistructured Interviews with Parents and Girls 


The semistructured interviews with program group girls focused on the girls’ experiences 
at PACE, what happened after their transitions from PACE, and their future plans. The interview- 
ers asked control group members about the services they received instead of PACE and their 
future plans. 


Questions in the parent or guardian interviews were similar in their aim to gain perspec- 
tive on PACE services received (or in the case of the control group parents, service contrast), the 
transition from PACE, and the parents’ perspectives on their daughters’ futures. However, the 
parent interviews focused more on how parents themselves had been involved in their daughters’ 
service receipt. 


Data Analysis 


Before the interviews, the study team developed key research questions and a plan for analysis. 
Upon completion of each interview, the interviewer detailed the participant’s responses in a stand- 
ard interview write-up form. The interview write-ups were exported into Excel workbooks, which 
organized responses by questions or topics covered within the interview and allowed for compar- 
ison across interviews. Exporting the data to Excel also allowed for cross-respondent comparison 
on questions with discrete responses, such as age of the respondent. The Excel workbooks were 
uploaded for further analysis in Dedoose, a secure web-based qualitative data coding program. 


To understand emergent themes in the qualitative responses, the analyst examined each 
excerpt in Dedoose, noting key themes. Codes were developed based on the themes that surfaced. 
These codes were applied to the data in Dedoose so that similar qualitative data extracts could be 
grouped by theme. This allowed the analyst to compare and contrast the data and draw conclu- 
sions to answer the research questions. Data from girls and from parents or guardians were ana- 
lyzed separately. 
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Appendix E 


Supplemental Tables for Chapter 2 


Appendix Table E.1 


Information on Source and Frequency of Service Receipt, 
Among Those Who Received the Service 


Outcome (%) 


Academic services 

Primary source of academic advising 
Parent, guardian, or other relative 
Someone from school 
PACE 
Other 


Frequency of academic advising sessions 
More than once per month 
Once per month 
1-3 times per year 
Never 


Primary source of help planning for a job or career 
Parent, guardian, or other relative 
Someone from school 
PACE 
Other 


Social services 
Primary source of help finding services in the community 
Public or alternative school 
Department of Children and Families 
Juvenile probation officer, Department of Juvenile Justice, 
or the court system 
Health or mental health provider 
PACE 
Other 


Primary counseling or therapy provider 
Health or mental health center, clinic, or private practice 
Public, alternative, or virtual school 
Department of Children and Families 
Department of Juvenile Justice, parole office, or the court system 
Community organization 
PACE 
Other 
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Program 
Group 


7.7 
8.5 
81.4 
2.4 


39.6 
19.1 
21.9 
19.3 


18.5 
5.8 
69.7 
6.0 


7.8 
8.0 


8.1 
8.3 
57.3 
10.6 


20.3 
2.8 
2.4 
5.17 
2.7 

64.2 
2.5 


Control 
Group 


41.2 
50.8 
0.9 
7.1 


33.1 
11.3 
25.2 
30.4 


58.3 
29.6 

0.4 
11.7 


34.8 
10.2 


26.9 
12.1 

1.7 
14.3 


53.0 
14.9 
5.8 
6.2 
8.7 
2.3 
9.2 


Difference 
(Impact) 


-33.4 
-42.3 
80.4 
-4.7 


6.5 
7.9 
-3.3 
-11.0 


-39.8 
-23.8 
69.3 
-5.7 


-27.1 
-2.2 


-18.9 
-3.8 
55.6 
-3.7 


-32.7 
-12.1 
-3.3 
-1.1 
-6.0 
61.8 
-6.7 


(continued) 


Appendix Table E.1 (continued) 


Program 
Outcome (%) Group 
Frequency of counseling or therapy sessions 
Once per week or more 38.4 
1-3 times per month 19.6 
Less than once per month 4.2 
Never 37.8 
Sample size (total = 863) 523 


SOURCE: MDRC calculations based on responses to the PACE evaluation 12-month follow-up survey. 


Control 
Group 


22.8 
13.4 

7.0 
56.7 


340 


Difference 
(Impact) 


15.5 
6.2 
-2.8 
-18.9 


NOTE: These measures are shown in italics because the analysis includes only those who reported receiving the 


service. Therefore, the analysis is not experimental. 
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Appendix Table E.2 


Source of Control Group Social Service Receipt 


Control 
Outcome (%) Group 
Source of help related to sexual and reproductive health 
Parent, guardian, or other relative 36.5 
Someone from school 27.5 
Health or mental health care provider 24.3 
Other 11.6 
Sample size 189 
Source of help related to social and emotional skills 
Parent, guardian, or other relative 39.4 
Someone from school 24.9 
Health or mental health care provider 17.4 
Other 18.3 
Sample size 213 


SOURCE: MDRC calculations based on responses to the PACE evaluation 12-month follow-up 
survey. 
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Appendix F 


Additional Information on Processing Florida 
Department of Education Records 


Variations in Time Frames 


The raw administrative records provided by the Florida Department of Education for the PACE 
evaluation aggregated data over time for each individual in the sample; however, the time period 
varied for different types of data. Because of this, outcomes may be presented for the calendar 
year, the academic year, or the summer term. For example, the raw records provided the total 
number of days enrolled over a full calendar year for each sample member, without distinguishing 
between days enrolled during the academic year and days enrolled in the summer. But the records 
provided the total number of credits earned over each semester for each sample member, making 
it possible to distinguish between credits earned in the academic year and credits earned in the 
summer term. 


Designation of Follow-Up Periods 


Data from the Florida Department of Education was available only by full semesters or full years, 
but sample members were randomly assigned throughout the school year and summer. Because 
of this, the 12-month follow-up period always aligns with the full school year, but it varies de- 
pending on when the sample members were randomly assigned. For participants randomly as- 
signed in June through December of a given year, the follow-up period is measured from the 
beginning of the school year (approximately mid-August). For those assigned in January through 
May, the follow-up period is measured from the beginning of the following school year. Appen- 
dix Figure F.1 illustrates this for girls randomly assigned during the 2014-2015 school year. 
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v6 


Random assignment 
month and year 


Appendix Figure F.1 


Follow-Up Period for Florida Department of Education Data, 
Based on Random Assignment Date 


2014 2015 


Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May 


2014 2015 2016 


Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul |!Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul 


Follow-up period Follow-up period 
School calendar year 2014-2015 School calendar year 2015-2016 


Appendix G 


Supplemental Impact Tables for Chapter 3 


Appendix Table G.1 
Additional One-Year Impacts on Risky Behavior and Mental Health 


Program Control Difference 
Outcome (%) Group Group (Impact) P-Value 


Sexual activity 
Used pregnancy protection method during 
last sexual encounter? 


Not sexually active or female partners only 61.2 64.1 -2.9 0.367 
Sexually active and used protection 29.9 27.3 2.6 0.394 
Sexually active and did not use protection 8.8 8.6 0.3 0.889 


Sexual activity in past 3 months 


Never sexually active 41.0 47.0 -6.0 * 0.061 
Previously sexually active, no partners in past 3 months 16.8 14.3 2.5 0.329 
Had one partner 33.6 32.7 0.9 0.780 
Had more than one partner 8.7 6.1 2.6 0.166 


Substance use 
Used marijuana since random assignment 38.2 35.1 3.1 0.348 


Mental health 
Experienced serious psychological distress” 14.0 15.3 -1.3 0.605 


Sample size 527 341 


SOURCE: MDRC calculations based on responses to the PACE evaluation 12-month follow-up survey. 


NOTES: Results in this table are regression-adjusted, controlling for pre-random assignment characteristics. 
Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

aMeasure includes only voluntary sexual encounters. "Not sexually active" means participant never had sex or 
had no partners in the past three months. 

bMeasure reflects distress in the previous 30 days, as assessed by the Kessler-6 scale. 
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Appendix Table G.2 


Additional 18-Month Impacts on Juvenile Justice Outcomes 


Program Control Difference 

Outcome (%) Group Group (Impact) 
Charge type 

Person 10.0 11.4 -1.4 

Property 9.8 9.5 0.3 

Drugs 2.1 2.0 0.0 

Public order 3.8 3.9 -0.1 

Violation of probation or prior sanction 9.3 10.2 -0.9 
Delinquent adjudication and adjudication withheld 
Offense types for delinquent adjudication 

Person/violent 1.3 1.8 -0.5 

Property 2.2 1.6 0.6 

Drugs 0.2 0.2 -0.1 

Public order 0.7 0.9 -0.2 

Violation of probation or prior sanction 3.5 2.7 0.8 
Category for delinquent adjudication 

Felony 1.8 1.4 0.4 

Misdemeanor 2.3 2.5 -0.1 

Other? 3.5 2.7 0.8 
Disposition/sanction for delinquent adjudication 

Out-of-home placement 2.6 1.8 0.8 

Community-based (including probation) 2.8 2.7 0.1 

Other type of sanction” 1.0 0.2 0.8 


Disposition/sanction for adjudication withheld 


Probation 8.3 9.2 -0.9 
Diversion program 0.4 1.1 -0.7 
Other type of sanction® 1.5 1.6 -0.1 
Civil citations 
Ever received a civil citation 3.7 3.8 0.0 
Ever completed a civil citation 3.4 3.6 -0.2 
Sample size 673 452 1,125 


98 


P-Value 


0.426 
0.884 
0.984 
0.919 
0.583 


0.485 
0.445 
0.814 
0.715 
0.425 


0.592 
0.890 
0.425 


0.371 
0.905 
0.098 


0.578 
0.157 
0.872 


0.967 


0.836 


(continued) 


Appendix Table G.2 (continued) 


SOURCE: MDRC calculations based on data from the Florida Department of Juvenile Justice (DJJ). 


NOTES: Results in this table are regression-adjusted, controlling for pre-random assignment characteristics. 
Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Rounding may cause slight discrepancies in sums and differences. 

a"Other" is the DJJ categorization for violations of probation or prior sanction. 

>Other types of sanction include judicial warning, other judicial disposition, and disposition transferred to another 
county. 

cOther types of sanction include judicial warning, other judicial disposition, disposition transferred to another 
county, teen court, and community arbitration mediation. 
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00I 


Outcome 


Academic engagement 
Number of unexcused absences in the past month 


Percentage of days absent in full calendar year 


Juvenile justice (%) 


Adjudicated delinquent since random assignment 


Sexual activity (% 


Sexually active and did not use pregnancy 
protection method during last sexual encounter? 


Social support and interpersonal relationships 
Has supportive adult in her life? (%) 


Mother relationship score® (average) 


Sample size 


Appendix Table G.3 
Impacts by Age 
Younger (11 to 14 Years Old) 


Program Control Difference 
Group Group _ (Impact) P-Value 


2.2 2.5 -0.3 0.589 
17.5 19.7 -2.2 0.260 

5.3 2.1 3.2 * 0.081 

3.3 4.7 -1.4 0.511 
60.3 59.1 1.2 0.824 
3.7 3.6 0.1 0.112 
288 175 


Older (15 to 17 Years Old) 


Program Control Difference 
Group Group _ (Impact) P-Value 


2.1 3.3 -1.1 ** 0.034 
19.7 24.8 -5.1 *** 0.006 
5.5 5.7 -0.2 0.901 
13.1 11.4 1.6 0.585 
61.5 62.2 -0.7 0.869 
3.7 3.7 0.0 0.863 
385 277 


Difference 
Between 
Subgroup 
Impacts 


SOURCE: MDRC calculations based on responses to the PACE evaluation 12-month follow-up survey, data from the Florida Department of Juvenile Justice, 


and data from the Florida Department of Education. 


NOTES: Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 
The H-statistic is used to assess whether the difference between subgroup impacts is statistically significant. Statistically significant differences between 

subgroups are indicated as follows: ttt = 1 percent, tt = 5 percent, t = 10 percent. 
aMeasure refers only to voluntary sexual activity in the past three months. 


>Other than a parent or guardian. 


°Score is calculated as an average of responses to 12 items on the Mother subscale of the Inventory of Parent and Peer Attachment. The possible score 


range is 1 to 5. 
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Outcome 


Academic engagement 


Number of unexcused absences in past month 


Appendix Table G.4 
Impacts by Prior Juvenile Justice Involvement 


No Involvement 


Program Control Difference 
Group Group _ (Impact) P-Value G 


Juvenile justice and self-reported delinquent behavior (%) 


Adjudicated delinquent 


Involved in any delinquent property incidents 


Involved in any delinquent violent incidents 


Sexual activity (% 


Sexually active and did not use pregnancy 
protection method during last sexual encounter® 


Social support and interpersonal relationships 


Has supportive adult in her life? (%) 


Mother relationship score® (average) 


Sample size 


2.1 2.9 -0.8 * 0.063 

1.5 0.6 0.9 0.249 
39.6 48.9 -9.3 ** 0.021 
20.5 23.8 -3.3 0.330 
6.6 6.2 0.4 0.838 
59.9 56.7 3.2 0.432 
3.7 3.7 0.0 0.567 
459 311 


roup 


2.5 


13.6 


41.6 


28.1 


13.9 


63.9 


3.8 


214 


Prior Involvement 


Group 


3.0 


12.7 
51.2 


29.3 


15.3 


72.0 
3.7 


141 


Program Control Difference 


(Impact) 


-1.5 


0.1 


P-Value 


0.497 


0.804 


0.144 


0.839 


0.747 


0.186 


0.395 


Difference 
Between 
Subgroup 
Impacts 


SOURCE: MDRC calculations based on responses to the PACE evaluation 12-month follow-up survey and data from the Florida Department of Juvenile 


Justice. 


NOTES: Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 
No statistically significant differences were found between subgroups on any impacts. 
aMeasure refers only to voluntary sexual activity in the past three months. 


>Other than parent or guardian. 


°Score is calculated as an average of responses to 12 items on the Mother subscale of the Inventory of Parent and Peer Attachment. The possible score 


range is 1 to 5. 
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Outcome 


Academic engagement 
Number of unexcused absences in past month 


Juvenile justice (%) 
Adjudicated delinquent 


Sexual activity (% 


Sexually active and did not use pregnancy 
protection method during last sexual encounter? 


Appendix Table G.5 


Impacts by Suspension and Expulsion History 


Social support and interpersonal relationships 


Has supportive adult in her life? (%) 
Mother relationship score® (average) 


Sample size 


Never Suspended or Expelled Ever Suspended or Expelled Difference 
Between 
Program Control Difference Program Control Difference Subgroup 
Group Group _ (Impact) P-Value Group Group (Impact) P-Value Impacts 
1.9 3.2 -1.3** 0.014 2.2 2.5 -0.3 0.620 
5.1 4.2 1.0 0.583 7.2 7.3 -0.1 0.984 
7.9 8.9 -1.0 0.712 10.3 8.7 1.6 0.658 
60.8 65.0 -4.2 0.400 60.8 56.8 4.0 0.512 
3.7 3.7 0.0 0.570 3.7 3.7 0.0 0.996 
349 217 227 145 


SOURCE: MDRC calculations based on responses to the PACE evaluation 12-month follow-up survey and data from the Florida Department of Juvenile 


Justice. 


NOTES: Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 
No statistically significant differences were found between subgroups on any impacts. 
aMeasure refers only to voluntary sexual activity in the past three months. 


>Other than parent or guardian. 


°Score is calculated as an average of responses to 12 items on the Mother subscale of the Inventory of Parent and Peer Attachment. The possible score 


range is 1 to 5. 


cOl 


Outcome 


Academic engagement 


Number of unexcused absences in past month 


Juvenile justice (%) 
Adjudicated delinquent 


Sexual activity (% 


Sexually active and did not use pregnancy 
protection method during last sexual encounter? 


Appendix Table G.6 
Impacts by History of Being Held Back a Grade 


Never Held Back a Grade 


Social support and interpersonal relationships 


Has supportive adult in her life? (%) 


Mother relationship score® (average) 


Sample size 


Program Control Difference Program Control 
Group Group (Impact) P-Value Group Group 
2.1 2.7 -0.6 0.249 2.0 3.0 

4.7 2.6 2.1 0.211 6.0 6.6 

6.4 8.2 -1.8 0.511 10.6 8.4 
64.2 59.4 4.8 0.357 57.3 64.0 
3.6 3.6 0.0 0.883 3.8 3.8 

314 196 330 218 


Ever Held Back a Grade 


Difference 
(Impact) 


-0.9 * 


-0.6 


2.2 


0.0 


P-Value 


0.093 


0.764 


0.456 


0.170 


0.919 


Difference 
Between 
Subgroup 
Impacts 


SOURCE: MDRC calculations based on responses to the PACE evaluation 12-month follow-up survey and data from the Florida Department of Juvenile 


Justice. 


NOTES: Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 
No statistically significant differences were found between subgroups on any impacts. 
aMeasure refers only to voluntary sexual activity in the past three months. 


>Other than parent or guardian. 


°Score is calculated as an average of responses to 12 items on the Mother subscale of the Inventory of Parent and Peer Attachment. The possible score 


range is 1 to 5. 


Appendix Figure G.1 

Distribution of Sample Members’ Race/Ethnicity by PACE Center 
Site A 
Site B 
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SOURCE: MDRC calculations based on data from the PACE management information system. 
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Appendix H 


Methodology for Calculating the Cost of PACE 


This appendix describes the process used to determine the costs of providing PACE services. 
PACE headquarters provided revenue and expenditure data for fiscal year 2015 (July 2014 to 
June 2015), which reflects a time period when PACE was enrolling girls in the study for the entire 
fiscal year. Financial data were also available at the local level for each center in the evaluation, 
but the research team determined that due to the circumstances of some centers, their reported 
expenses were not a true reflection of their total costs. For example, some centers do not pay rent 
for their buildings, so facilities costs were not captured in their expenses; other centers offered 
services beyond the day program that is the subject of the current evaluation. ! 


Therefore, PACE costs were determined using a combination of implementation data 
gathered for the study and financial information reported by PACE. The research team used a 
two-step process to estimate center-level costs.” First, implementation data were used to specify 
a comprehensive list of ingredients needed to implement the PACE model each year. The main 
categories of ingredients include staff and benefits, facilities, program supplies and supports (text- 
books, food, student incentives, and so on), transportation, and overhead costs (including mainte- 
nance, utilities, and insurance, as well as fundraising costs). The categories of ingredients were 
further refined using PACE’s chart of accounts, which provides descriptions for each revenue and 
expense category on PACE’s balance sheets. 


The second step was to assign annual prices to each ingredient. PACE’s financial data 
were used to create the prices. Though the PACE model is very similar across the sites, there are 
differences in aspects of the locations and the centers’ structures that lead to variation in the costs. 
As indicated above, rent is one area where center costs can vary quite a bit. The 14 centers in the 
study had an average annual rent of $64,000, but actual annual rents ranged from $0 to $156,000. 
Centers also vary in size, with most centers serving around S50 girls at a time, but centers in the 
study ranged from 42 slots up to 82 slots in fiscal year 2015. Capacity drives considerable varia- 
tion in staffing costs as well, with an average of about $1,010,000 in staffing and benefits costs 
(range: $672,000-$2,011,000) for the 14 centers in the study. Even centers with similar planned 
staffing levels had variation in annual staffing costs, depending on whether the center had staff 
members of longer tenure, who are more expensive, or lengthy staff vacancies, which can reduce 
costs in the short term. Because of this variation in costs across the centers, the evaluation team 
elected to use the cost data from a sample of “average” centers to create the annual prices for each 
ingredient. The evaluation team selected six centers that reflected the average center size and 
standard staffing arrangement. These centers had an average size of 51 slots and 16.5 full-time- 
equivalent staff members. 


The annual price for each ingredient was calculated by averaging financial data from the 
six centers for fiscal year 2015. When one of the six centers had unusual circumstances that would 


‘Simply taking PACE’s overall FY 2015 budget of $29 million for the 14 centers in the study and headquar- 
ters and dividing it by the total number of slots and average length of stay yields a cost per girl of $24,271. This 
number is slightly higher than our estimate, likely because it includes costs of services beyond the day program 
at some centers, and headquarters costs associated with running five additional PACE centers that year that were 
not part of the study. 


The approach taken generally follows the ingredients method described by Levin et al. (2017). 
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artificially lower the cost estimate, such as paying no rent or receiving materials for free, that 
center was excluded from the average cost calculations for that component. In those cases, the 
costs for the remaining centers were averaged to calculate the price. The financial data did allow 
the research team to account for some in-kind services in the calculations. For the most part, in- 
kind services consisting of donations of goods or services for PACE centers to use as part of the 
student incentive program or for fundraising activities were tracked on the financial statements. 
The contribution of volunteer hours was not included in the financial data, but the implementation 
data found that volunteers did not play a major role in providing program services. Interns did 
play a significant role in delivery of social services at many centers at the time of the researchers’ 
site visits, but their hours were not tracked in the financial information. The annual value of a 
half-time graduate-level social work intern is included in the estimates. 


The annual prices for overhead costs, which include occupancy, utilities, supplies, equip- 
ment, and insurance, were created from average reported expenditures across the six selected 
centers. The overhead costs were then allocated across the functional categories described in 
Chapter 4. Allocations of overhead costs by functional category were based primarily on the space 
required to provide each service. The academic program, which has the largest physical footprint 
at a PACE center, was allocated 60 percent of overhead costs, social services received 30 percent, 
and follow-up services and outreach and recruitment received 5 percent each. Costs related to 
fundraising were allocated to functional categories in the same fashion. 


PACE headquarters costs are also included in the average center costs. The implemen- 
tation research found that the PACE program model was uniformly implemented across sites, 
in large part due to the support provided by headquarters. PACE headquarters provides many 
services to the centers, and its costs include headquarters-based staffing costs, staff training and 
training-related travel, staff recruitment, information technology, human resources and payroll 
functions, and other insurance and legal costs. To generate a per-center headquarters cost, the 
total headquarters cost for fiscal year 2015 was divided by 19, the number of PACE centers 
operating during that time. The headquarters cost was allocated across functional categories 
using the same proportions as for overhead costs. Headquarters costs add about $346,000 per 
center per year. 


The annual operating cost per center was determined by applying a cost to each ingredi- 
ent, then summing across all ingredients. This cost per center serves as the basis for the cost per 
girl calculation. Though the average center on which these calculations are based can serve 51 
girls at a time, the centers serve more girls over the course of a year because girls stay about 7.9 
months on average. The estimates presented here assume that the center was operating at full 
capacity during the entire year, which was true for most of the centers in the evaluation. Though 
centers occasionally had to suspend random assignment because they did not have excess appli- 
cants to fill the control group, most centers were at or near capacity most of the time. The cost per 
girl is calculated by dividing the annual center-level component cost by 51, the number of slots 
available at one time, and dividing that by 12 to arrive at a monthly cost per slot, then multiplying 
that number by 7.9 (the average number of months a girl is active in the program). 
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About MDRC 


MDRC is a nonprofit, nonpartisan social and education policy research organization dedicated 
to learning what works to improve the well-being of low-income people. Through its research 
and the active communication of its findings, MDRC seeks to enhance the effectiveness of so- 
cial and education policies and programs. 


Founded in 1974 and located in New York; Oakland, California; Washington, DC; and Los 
Angeles, MDRC is best known for mounting rigorous, large-scale, real-world tests of new and 
existing policies and programs. Its projects are a mix of demonstrations (field tests of promising 
new program approaches) and evaluations of ongoing government and community initiatives. 
MDRC’s staff members bring an unusual combination of research and organizational experi- 
ence to their work, providing expertise on the latest in qualitative and quantitative methods and 
on program design, development, implementation, and management. MDRC seeks to learn not 
just whether a program is effective but also how and why the program’s effects occur. In addi- 
tion, it tries to place each project’s findings in the broader context of related research — in order 
to build knowledge about what works across the social and education policy fields. MDRC’s 
findings, lessons, and best practices are shared with a broad audience in the policy and practi- 
tioner community as well as with the general public and the media. 


Over the years, MDRC has brought its unique approach to an ever-growing range of policy are- 
as and target populations. Once known primarily for evaluations of state welfare-to-work pro- 
grams, today MDRC is also studying public school reforms, employment programs for ex- 
prisoners, and programs to help low-income students succeed in college. MDRC’s projects are 
organized into five areas: 

e Promoting Family Well-Being and Children’s Development 

e Improving Public Education 

e Raising Academic Achievement and Persistence in College 

e Supporting Low-Wage Workers and Communities 


e Overcoming Barriers to Employment 


Working in almost every state, all of the nation’s largest cities, and Canada and the United 
Kingdom, MDRC conducts its projects in partnership with national, state, and local govern- 
ments, public school systems, community organizations, and numerous private philanthropies. 


